Develop Reference

How to stop threads created by org.asynchttpclient to prevent memory leak?

I am using org.asynchttpclient to post async request.
While shutting down tomcat, I am getting below logs:
SEVERE: The web application [/test] appears to have started a thread named [pool-1-thread-1] but has failed to stop it. This is very likely to create a memory leak.
Jul 04, 2017 10:53:00 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/test] appears to have started a thread named [AsyncHttpClient-2-1] but has failed to stop it. This is very likely to create a memory leak.
Jul 04, 2017 10:53:00 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/test] appears to have started a thread named [AsyncHttpClient-2-2] but has failed to stop it. This is very likely to create a memory leak.
Jul 04, 2017 10:53:00 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/test] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#71513074]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap#4aecc2e8]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
How to close these threads to prevent memory leak.

This is really more of a warning and in most times can be ignored. You will see this error when the context is reloaded or shut down. If you are shutting down the server then the JVM will be shutdown also and the os will clean up any memory.
If you want to try and call close on the AsyncHttpClient which implements Closable it may stop this issue. You will probably have to put a wait in there to let it fully close the connection factory. You can look at the tomcat Lifecycle to add a shutdown handler to handle closing the client.

Tomcat crashes while access large database table

When my database request takes too long tomcat crashes / closes and I find this in my log. I am totally clueless right now :)
Any thoughts ?
SEVERE: The web application [/ir] is still processing a request that
has yet to finish. This is very likely to create a memory leak. You
can control the time allowed for requests to finish by using the
unloadDelay attribute of the standard Context implementation.
Mar 14, 2015 2:14:38 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/ir] appears to have started a thread
named [MySQL Statement Cancellation Timer] but has failed to stop it.
This is very likely to create a memory leak.
What should I be doing ?

Spring JPA unknown error but application works as expected

I'm using Spring 4.0, tomcat 8, spring-data-jpa 1.7.0.RELEASE, hibernate 4.1, mysql 5.6.21 Homebrew. I'm new in java, just moved form C++ background. I assume there must be some resource leaking. I have to close or free something before I exist the app. But no idea what to do. Please advise me!
Every time I restart I got following error:-
Nov 20, 2014 2:01:01 PM org.apache.catalina.loader.WebappClassLoaderBase
clearReferencesThreads
WARNING: The web application [MyJap] appears to have started a thread named [MySQL
Statement Cancellation Timer] but has failed to stop it. This is very likely to create a
memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.util.TimerThread.mainLoop(Timer.java:526)
java.util.TimerThread.run(Timer.java:505)
Nov 20, 2014 2:01:01 PM org.apache.coyote.AbstractProtocol start

Tomcat Memory Leak Using Struts2 xwork

SEVERE: The web application [/ppAdmin] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10#126212dd]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;#64c47498]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Mar 22, 2012 12:43:00 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/ppAdmin] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10#1dc2dad7]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;#63ad6884]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Mar 22, 2012 12:43:00 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/ppAdmin] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10#1dc2dad7]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;#da8e53c]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Mar 22, 2012 12:43:01 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/Nemo] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Mar 22, 2012 12:43:01 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/Nemo] registered the JDBC driver [net.sourceforge.jtds.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Mar 22, 2012 12:43:01 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/Nemo] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak.
Mar 22, 2012 12:43:01 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/Nemo] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10#28318d]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;#4ecd200f]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Mar 22, 2012 12:43:01 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/Nemo] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10#7cf3e133]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;#6f461b5b]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Mar 22, 2012 12:43:01 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/Nemo] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10#28318d]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;#63ae2c6f]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Mar 22, 2012 12:43:01 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/Nemo] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10#28318d]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;#3fe75e89]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Mar 22, 2012 12:43:01 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/Nemo] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10#28318d]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;#6eef9525]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Mar 22, 2012 12:43:01 PM org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8080
WHY THE MEMORY LEAK WARNINGS ARE THERE???TOMCAT 6.0.35 ALLOWS GC TOOL.STILL IT GIVES SUCH WARNINGS.I AM BOTHERED ABOUT THESE WARNINGS.PLEASE HELP ME HOW TO MANAGE WITH THESE?
ANY HELP WILL BE HIGHLY APPRECIATED..................THANKS IN ADVANCE.

I had the same problem with Tomcat 6 and Struts2 2.1.8 webapp (maven 3.0.3 + eclipse keepler + JDK 1.6.0_22). I'm new in struts2+tomcat developing and I'm not used to solve this kind of problems.
After having made some changes on my code (and don't use svn nor git...), and after start using a mvn tomcat:redeploy task (witch confused me) I had this same behavior, and don't know what were causing to fail the deployment (the project keeps compiling ok).
The error seems so confusing and zero descriptive to me, and I've read many related posts. I've tryed to update the struts2 version, cause I've read that this kind of problems seems solved in struts2 2.3.3 (was using 2.1.8).
https://issues.apache.org/jira/browse/WW-3768
I dont't know why I was just reading catalina log:
SEVERE: Error filterStart
12/02/2014 10:46:23 org.apache.catalina.core.StandardContext start
SEVERE: Context [/validador] startup failed due to previous errors
The answer was very close, in localhost-date.log!
That point me to a bad declared action in struts.xml, so simple to solve and so descriptive, but so dificult to figure out just reading catalina log.
SEVERE: Exception starting filter struts2
Unable to load configuration. - action - file:/C:/apps/apache-tomcat- 6.0.39/webapps/validador/WEB-INF/classes/struts.xml:74:83
at org.apache.struts2.dispatcher.Dispatcher.init(Dispatcher.java:501)
at org.apache.struts2.dispatcher.ng.InitOperations.initDispatcher(InitOperations.java:74)
at org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.init(StrutsPrepareAndExecuteFilter.java:57)
...
...
Caused by: Action class [aoc.action.signature.CompletePDFUpload] not found - action - file:/C:/apps/apache-tomcat-6.0.39/webapps/validador/WEB-INF/classes/struts.xml:74:83
I hope someone else missed to check the correct log and solve as easy as me their problem.

Tomcat 6 memory leaks log entries

Below is outtake of unique entries in my Catalina.out file on CentOS machine. I'm running Tomcat 6 with spring 3 and my application. There is whole bunch of them so I just picked some that keep repeating. This doesn't happen all the time but it happens at least once a week.
The Question is what can I do to prevent the bellow from happening?
Feb 3, 2011 2:37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [] registered the JBDC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Feb 3, 2011 2:37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [com.iteezy.shared.domain.DirEntry.data] but has failed to stop it. This is very likely to create a memory leak.
Feb 3, 2011 2:37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named
[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0] but has failed to stop it. This is very likely to create a memory leak.
Feb 3, 2011 2:37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [File Reaper] but has failed to stop it. This is very likely to create a memory leak.
Feb 3, 2011 2:37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [pool-1-thread-22] but has failed to stop it. This is very likely to create a memory leak.
37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
b application [] appears to have started a thread named
[org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
37:48 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
b application [] created a ThreadLocal with key of type [net.sf.json.AbstractJSON$1] (value [net.sf.json.AbstractJSON$1#40bbb3d6]) and a value of type [java.util.HashSet] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

When you define an external flag which a thread is suppose to poll and exit when it's set - it must be volatile. Otherwise the thread might never see the change made by other thread.
However there is already a feature like that in standard API - it is called an interrupt() method and Thread.currentThread().isInterrupted(). No need to duplicate already existing logic. See: Stopping a specific java thread.
That being said calling interrupt() on each and every thread is a bad idea as well because there is no guarantee that all threads respond to it. Examining your exceptions I noticed the following threads not being cleaned up properly:
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0 - close the C3P0 data source. Since you are using Spring simply add destroy-method="close". We're done with this thread.
File Reaper - as far as I can see this thread is created by FileCleaningTracker. You need to call FileCleaningTracker.exitWhenFinished() explicitly when shutting down your application (or when the class is no longer needed, I never used it) or let Spring to do this (see above). Chances are some 3rd party library uses it and not closing properly - this means it has a bug.
pool-1-thread-22 - this is one of the threads created by Executors utility inside ExecutorService. Make sure you call shutdown() on every such a pool in your application during shutdown.
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-2 - Quartz worker thread (the one that actually runs jobs). SchedulerFactoryBean closes the scheduler for you automatically, I think Tomcat is mistaken here, I see this error often as well. Nevertheless looks like setting SchedulerFactoryBean.waitForJobsToCompleteOnShutdown to true solves this.
com.iteezy.shared.domain.DirEntry.data - I am not sure about this one. It is either your own thread that needs to be interrupted upon shutdown or H2 database thread (?) Its stack needs to be examined to guess where does it come from.
The bottom line is: don't just kill everything that moves (actually, Tomcat does that for you after issuing this warning) but determine where the threads come from and use framework/library specific close() method to allow further cleanup. Be gentle.

Set up a Servlet to manage this in its destroy() method. The threads can check a flag to see if they must continue or not.
In your servlet, do the following in the destroy method. You will obviously need to be able to have access to a Collection<MyThread> but how you get that really depends on how your system is set up.
destroy() {
for (MyThread thread : myThreads) {
thread.stopProcessing();
}
}
Your MyThread class will have something like this:
public class MyThread {
private boolean finished = false;
#Override
public void run() {
while (!finished) {
//do something
}
}
public void stopProcessing() {
finished = true;
}
}

Shutdown your webapp properly. Don't leave these threads running.
Alternatively, don't keep redeploying the webapp.