Develop Reference

Cassandra Node refuses to Join Cluster "Compaction Executor" error

We have a 3 node Cassandra Cluster running the following version
[cqlsh 5.0.1 | Cassandra 3.11.6 | CQL spec 3.4.4 | Native protocol v4]
Node1 stopped communicating with the rest of the cluster this morning, the logs showed this:
ERROR [CompactionExecutor:242] 2020-09-15 19:24:48,753 CassandraDaemon.java:235 - Exception in thread Thread[CompactionExecutor:242,1,main]
ERROR [MutationStage-2] 2020-09-15 19:24:54,749 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[MutationStage-2,5,main]
ERROR [MutationStage-2] 2020-09-15 19:24:54,771 StorageService.java:466 - Stopping gossiper
ERROR [MutationStage-2] 2020-09-15 19:24:56,791 StorageService.java:476 - Stopping native transport
ERROR [CompactionExecutor:242] 2020-09-15 19:24:58,541 LogTransaction.java:277 - Transaction log [md_txn_compaction_c2dbca00-f780-11ea-95eb-cf88b1cae05a.log in /mnt/cass-a/data/system/local-7ad54392bcdd35a684174e047860b377] indicates txn was not completed, trying to abort it now
ERROR [CompactionExecutor:242] 2020-09-15 19:24:58,545 LogTransaction.java:280 - Failed to abort transaction log [md_txn_compaction_c2dbca00-f780-11ea-95eb-cf88b1cae05a.log in /mnt/cass-a/data/system/local-7ad54392bcdd35a684174e047860b377]
ERROR [CompactionExecutor:242] 2020-09-15 19:24:58,566 LogTransaction.java:225 - Unable to delete /mnt/cass-a/data/system/local-7ad54392bcdd35a684174e047860b377/md_txn_compaction_c2dbca00-f780-11ea-95eb-cf88b1cae05a.log as it does not exist, see debug log file for stack trace
Cassandra starts up fine on the "broken node", but refuses to rejoin the cluster.
When I do a nodetool status I get this:
**Error: The node does not have system_traces yet, probably still bootstrapping**
Gossip is not running, i've tried disabling and re-enabling, no joy.
I've also tried both a repair and a rebuild, both came back with no errors at all.
Any and all help would be appreciated.
Thanks.

The symptoms you described indicates to me that the node had some form of hardware failure and the data/ disk is possibly inaccessible.
In instances like this, the disk failure policy in cassandra.yaml kicked in:
disk_failure_policy: stop
This would explain why gossip is unavailable (on default port 7000) and the node would not be accepting any client connections either (on default CQL port 9042).
If there is an impending hardware failure, there's a good chance the disk/volume is mounted as read-only. There's also the possibility that the disk is full. Check the operating system logs for clues and you will likely need to escalate the issue to your sysadmin team. Cheers!

OpenAM13.0.0--Tomcat Shutdown Automatically

We have deployed openam13.0.0 war in Tomcat8.026 version server. If server is idle for 12 to 24 hours, it gets stopped automatically and throws the below error. Please provide your suggestions on this.
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
The web application [openam] appears to have started a thread named
[com.google.inject.internal.util.$Finalizer] but has failed to stop
it. This is very likely to create a memory leak. Stack trace of
thread: java.lang.Object.wait(Native Method)

This is related to Tomcat:
Guice + Tomcat potential memory leak
and more than likely this bug:
https://bugster.forgerock.org/jira/browse/OPENAM-3133
The issue you are seeing is happening only when Tomcat is stopping. As long as Tomcat isn't actually prevented from shutting down then these are fairly harmless.
As reported in the bug report, this issue has been there since OpenAM 10 with recent version of Tomcats, it is fair to assume that the errors can be safety ignored as long as the Tomcat is able to shutdown. The discussion in the bug report OpenAM-3133 give a fairly good assessment of the situation.
Cheers

java.lang.IllegalStateException while reloading Tomcat

I'm working on Spring with Hibernate application, It's working fine but while starting or reloading tomcat server, i'm getting java.lang.IllegalStateException Exception.
Can anybody explain, why this exception occurring and how to resolve it??
INFO: Illegal access: this web application instance has been stopped already. Could not load java.net.BindException. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
at com.mysql.jdbc.SQLError.createLinkFailureMessageBasedOnHeuristics(SQLError.java:1220)
at com.mysql.jdbc.exceptions.jdbc4.CommunicationsException.<init>(CommunicationsException.java:57)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3270)
at com.mysql.jdbc.MysqlIO.quit(MysqlIO.java:1659)
at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4296)
at com.mysql.jdbc.ConnectionImpl.cleanup(ConnectionImpl.java:1265)
at com.mysql.jdbc.ConnectionImpl.finalize(ConnectionImpl.java:2667)
at java.lang.System$2.invokeFinalize(Unknown Source)
at java.lang.ref.Finalizer.runFinalizer(Unknown Source)
at java.lang.ref.Finalizer.access$100(Unknown Source)

I want to give you couple of choices. You can try it. Any option can fulfill your demand.
Restart your tomcat and apache server because a long time of using,
it keeps older version of your application.
Clean your tomcat temp directory and restart
As stated in error,
The eventual following stack trace is caused by an error thrown for
debugging purposes as well as to attempt to terminate the thread which
caused the illegal access, and has no functional impact.
It may be the real cause that at this moment you are in debugging mode and it doesn't clear the running thread. so remove your break point and run it not debugging
If your code contains any kind of thread which are not properly terminated, then this type of error may occur.
That you have init() method but no destroy() method, then this type of error may occur. For details, you can follow the link - http://www.javaspecialists.eu/archive/Issue056.html
if the webapp has stopped, or is stopping, that means that the .war file or WEB-INF/web.xml timestamp changed, and the webapp might be reloading. Please check timestamp is OK or not.
To turn it off, set reloadable="false" in the context definition of
your app. It might be tomcat's server.xml.
In details solution:
The tomcat's server.xml of the reloadable Context is set to false.
For example:
Context path="/expert" docBase="expert" debug="0" reloadable ="false"/>
The solution is easy, as long as the tomcat's server.xml in reloadable = "true" into false on the line, but to do so would lose the advantage of hot deployment, and for the development is not very convenient, simply change it or not. This error does not matter.
Error principle:
The reason is because the tomcat restart, because the previous tomcat thread has not completely shut down, restart tomcat will report this exception, but this does not affect the normal use, just jump abnormal annoying people. Used hibernate, spring or other large components, when a WEB application system has a lot of class,
if you turned the Tomcat reloadable = true, then whenever the relevant documents change, Tomcat stops web app and frees up memory, and then reload web app. it may be a huge project. So we always think if there is only a certain class of overloaded functions, will greatly meet our debugger.
UPDATE: For code related issue
First, I want to tell you that I have given you some solutions for tomcat basis. Now I want to give you a solution for code basis. Would you please cross check your code with this issue? Please follow the URL.
http://www.coderanch.com/t/660126/Wiki/Illegal-State-Exception
UPDATE: For MySQL related issue
There are 2 issues.
This web application instance has been stopped already. Could not load java.net.BindException.
This web application instance has been
stopped already.Could not load com.mysql.jdbc.
This is because the MySQL JDBC driver on the application under the WEB-INF/lib directory, in the re-release of its loaded twice, so long as it can be copied to %TOMCAT_HOME%/lib can solve the problem.
We can solve these two anomalies MySQL drivers from the WEB-INF/lib folder moved to %TOMCAT_HOME%/lib.
I suspect this might be happening after the web application is restarted, where it's down for a short period of time. Then some finalize() method in the code is probably trying to do some cleanup too late. Whether or not that's in your code or the MySQL driver I can't say. You definitely should only have one version of a jar in a directory at a time. You might want to upgrade it to the latest (5.1.38 right now) in case something has been fixed that might be affecting you.(Number 9 is copied from #WhiteFang34)
Related link for 9: tomcat 6.0.24 Exception: Could not load com.mysql.jdbc.SQLError

The issue was because of the server cache. Follow the below steps to fix this issue
Find the directory called "work" under your tomcat's root directory
Delete it completely
Restart the server
Issue will be fixed.
Note: In case you have default tomcat installation in linux machine, you have to delete the folder /var/cache/tomcat7/Catalina

I think that the root cause of this problem is that something is leaking JDBC Connection objects.
These objects are actually instances of com.mysql.jdbc.ConnectionImpl ... which is a class with a finalize(). Normally, object finalization tidies up the leaked connections. In this case, it looks like the sequence of events is as follows:
the GC runs after the webapp has shutdown
a ConnectionImpl object is found to be unreachable
the object's finalize() method attempts to shutdown the MySQL database connection ... which entails sending a message to the server
when the message send fails for some reason, and the comms-level code tries to create an exception by calling com.mysql.jdbc.SQLError.createCommunicationsException
that tries to create an exception reflectively ...
The last step is failing because it is trying to use the webapp's classloader for the webapp ... which has been shut down. Apparently, this is detected by a check that is intended to help programmers diagnose webapps that don't shutdown cleanly.
The best solution is to track down the source of the Connection leak(s) and fix it. If Connection objects did not leak, then they would be closed while the webapp's classloader was still active.
Alternatively, you could simply turn off the check, as described in #Skywalker's answer.

The problem could be a simple SQL Error. tThe
java.lang.IllegalStateException
means that an object or class was called on when it was not ready to be called on. These errors at the bottom of the stack
at java.lang.ref.Finalizer.runFinalizer(Unknown Source)
at java.lang.ref.Finalizer.access$100(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
look like they come from the Java Garbage Collector
attempting to clean up MySQL Object Pointers
that were temporarily unreferenced during a reboot. It seems to me that your error here is just a few MySQL Object Pointers being halted in the middle of execution when the GC calls their finalize methods and then pauses the application to attempt a clean up the object. (leading to a fail and a throw of this exception)
This error is most likely harmless, and could maybe be fixed by adding a
JNI_DestroyJavaVM();
to your main thread to be executed on shutdown.

java.lang.IllegalStateException. Possible error with WebClassLoader

I'm developing a web application with Spring and I have this strange error. I can't figure out where it comes from, I've been google'ing hard and found nothing so far.
In the project I'm using: MongoDB, Spring, WSS4J (X.509 security headers for soap), Apache CXF. I've been trying to discover whether there is something that causes this error, any particular action, behaviour, condition something, but I found nothing so far, sometimes it happens after 3 min of running sometimes after 3h or not at all, any clues and hints appreciated.
INFO: Illegal access: this web application instance has been stopped already. Could not load net.sf.ehcache.store.disk.Segment$1. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1597)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1556)
at net.sf.ehcache.store.disk.Segment$HashIterator.<init>(Segment.java:988)
at net.sf.ehcache.store.disk.Segment.hashIterator(Segment.java:936)
at net.sf.ehcache.store.disk.DiskStore$HashIterator.<init>(DiskStore.java:1038)
at net.sf.ehcache.store.disk.DiskStore$KeyIterator.<init>(DiskStore.java:1111)
at net.sf.ehcache.store.disk.DiskStore$KeyIterator.<init>(DiskStore.java:1111)
at net.sf.ehcache.store.disk.DiskStore$KeySet.iterator(DiskStore.java:949)
at net.sf.ehcache.store.disk.DiskStorageFactory$DiskExpiryTask.run(DiskStorageFactory.java:838)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)`

It means the ehcache thread is trying to interact with the application after the application has been stopped or redeployed. It happens only when ehcache reach limits you have in ehcache.xml and trying to overflow to disk.
Look at tomcat logs for warnings when stop your application:
SEVERE: The web application [/APP] appears to have started a thread
named [Ehcache_Worker-1] but has failed to stop it.
This is very likely to create a memory leak
If find something like this, you should stop ecache properly in your ServletContextListener:
public void contextDestroyed(ServletContextEvent servletContextEvent){
CacheManager.getInstance().shutdown();
}
maybe with 1 sec sleep after that, to be sure ehcache is stopped.
As mentioned in http://ehcache.org/documentation/faq
If the JVM keeps running after you stop using Ehcache, you should call
CacheManager.getInstance().shutdown() so that the threads are stopped
and cache memory is released back to the JVM. Calling shutdown also
insures that your persistent disk stores get written to disk in a
consistent state and will be usable the next time they are used. If
the CacheManager does not get shut down, it should not be a problem.
There is a shutdown hook which calls the shutdown on JVM exit.
Or you can set property overflowToDisk="false" in ehcache.xml or restart tomcat every time you deploy apllication.

How to start and stop Tomcat cleanly from Java and why my way doesn't work repeatably?

I am writing a Quartz application that runs on Windows and calls Lucene and Solr to run indexing jobs. It runs a sequence of jobs, and each job consists of these steps:
Make sure Tomcat is stopped (Solr running under Tomcat prevents index dir from being deleted or copied)
Delete old index directory if necessary
Start Tomcat
Make sure Tomcat and Solr app are running
Run the indexing job
Stop Tomcat
Make sure Tomcat is stopped
Copy index directory to an archive
I decided to have the code that starts and stops Tomcat set the system properties that are set in Startup.bat, Shutdown.bat and Catalina.bat, and just call Bootstrap.main with "start" and "stop" parameters. This worked for one iteration, but not when I tried a Quartz run in which I set up two iterations.
When my code shut down Tomcat at the end of the first iteration, all of the usual messages were displayed, including
INFO: Stopping ProtocolHandler ["http-bio-5918"]
(I am using port 5918) but when it tried to start Tomcat at the beginning of the second iteration, it got thes errors:
SEVERE: Failed to initialize end point associated with ProtocolHandler ["http-bio-5918"]
java.net.BindException: Address already in use: JVM_Bind <null>:5918
and
SEVERE: Failed to initialize connector [Connector[HTTP/1.1-5918]]
org.apache.catalina.LifecycleException: Failed to initialize component [Connector[HTTP/1.1-5918]]
I ran netstat -an in a command prompt window, and it confirmed that port 5918 was in use. There's nothing special about the code I am using to check if Tomcat is running. I've seen in various places on the internet.
public boolean isTomcatRunning(String url) {
boolean isRunning = false;
try {
new URL(url).openConnection().connect();
isRunning = true;
} catch (IOException e) {
isRunning = false;
}
return isRunning;
}
but it apparently tells me that Tomcat is not running when it is.
As I said, I am starting and stopping Tomcat by calling Bootstrap.main(new String[]{"start"}) and Bootstrap.main(new String[]{"stop"}). The only thing peculiar about that is that is that when I simply call Bootstrap.main(new String[]{"start"}), it doesn't seem to return (I haven't waited long enough yet to see if it is hanging or just taking a long time), so I have been running it inside a thread.
Maybe that is causing the problem, as it looks like Catalina.bat isn't doing anything special and it returns from startup just fine. I wonder if there is an additional setup I need to do to enable it to run startup in the main thread without hanging.
In any case, this is what I am puzzled about with starting and stopping Tomcat from within my Quartz application, and I would appreciate any help and suggestions you can offer.

I strongly suggest that you wrap your Tomcat instance with a wrapper that controls the lifecycle of your Tomcat instance. Such wrapper is the Java Service Wrapper. An older version (3.2.3 I believe) is "free" and works fine with newer Tomcat instances.
Your controlling application then "talks" to the wrapper to start/stop the Tomcat application. There are multiple benefits with this approach. One of them is that you are not subject to your Tomcat application hanging and the port you are testing not replying anymore.