I have a Spring web application running on Tomcat.
When trying to shutdown Tomcat with the shutdown.sh script, the java process doesn't end because it has a Thread that's still running. The catalina.log contains
Jul 22, 2013 2:07:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/myapp] appears to have started a thread named [com.google.common.base.internal.Finalizer] but has failed to stop it. This is very likely to create a memory leak.
Apparently BoneCP has some Guava integration that causes this thread to block. Is there a clean or intended way to kill this thread?
There's an open issue on Guava for this problem - the issue itself is pretty old but if you check the latest comments you'll see some suggestions from other BoneCP users to resolve this issue
Also check this other issue and this patch
Related
I am using org.asynchttpclient to post async request.
While shutting down tomcat, I am getting below logs:
SEVERE: The web application [/test] appears to have started a thread named [pool-1-thread-1] but has failed to stop it. This is very likely to create a memory leak.
Jul 04, 2017 10:53:00 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/test] appears to have started a thread named [AsyncHttpClient-2-1] but has failed to stop it. This is very likely to create a memory leak.
Jul 04, 2017 10:53:00 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/test] appears to have started a thread named [AsyncHttpClient-2-2] but has failed to stop it. This is very likely to create a memory leak.
Jul 04, 2017 10:53:00 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/test] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#71513074]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap#4aecc2e8]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
How to close these threads to prevent memory leak.
This is really more of a warning and in most times can be ignored. You will see this error when the context is reloaded or shut down. If you are shutting down the server then the JVM will be shutdown also and the os will clean up any memory.
If you want to try and call close on the AsyncHttpClient which implements Closable it may stop this issue. You will probably have to put a wait in there to let it fully close the connection factory. You can look at the tomcat Lifecycle to add a shutdown handler to handle closing the client.
When my database request takes too long tomcat crashes / closes and I find this in my log. I am totally clueless right now :)
Any thoughts ?
SEVERE: The web application [/ir] is still processing a request that
has yet to finish. This is very likely to create a memory leak. You
can control the time allowed for requests to finish by using the
unloadDelay attribute of the standard Context implementation.
Mar 14, 2015 2:14:38 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/ir] appears to have started a thread
named [MySQL Statement Cancellation Timer] but has failed to stop it.
This is very likely to create a memory leak.
What should I be doing ?
I have been searching a lot but have not found a perfect one which can help me to solve this problem.
I have web services and am generating the stub using JAX-WS.
To access the methods of web services I have wriiten a class in which all the methods are static like
public static String getLocation()
{
//call to the web service
}
I am specifying static because I want to confirm this is not the root cause of my problem.
Now when I am checking the logs in the Tomcat directory the catilina log shows some thins like this...This error is occured when I Startup or shutdown the tomcat server
Mar 18, 2010 11:13:07 PM org.apache.catalina.core.ApplicationContext log
INFO: HTMLManager: stop: Stopping web application at '/testWeb'
Mar 18, 2010 11:13:07 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: A web application appears to have started a thread named [leakingThread] but has failed to stop it. This is very likely to create a memory leak.
Another error I am seeing is
SEVERE: Unable to determine string representation of value of type [com.sun.xml.stream.writers.XMLStreamWriterImpl]
java.lang.UnsupportedOperationException
at com.sun.xml.stream.writers.XMLStreamWriterImpl.entrySet (XMLStreamWriterImpl.java:2134)
at java.util.AbstractMap.toString(AbstractMap.java:478)
at org.apache.catalina.loader.WebappClassLoader.clearThreadLocalMap(WebappClassLoader.java:2433)
at org.apache.catalina.loader.WebappClassLoader.clearReferencesThreadLocals(WebappClassLoader.java:2349)
at org.apache.catalina.loader.WebappClassLoader.clearReferences(WebappClassLoader.java:1921)
at org.apache.catalina.loader.WebappClassLoader.stop(WebappClassLoader.java:1833)
at org.apache.catalina.loader.WebappLoader.stop(WebappLoader.java:740)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4920)
at org.apache.catalina.core.ContainerBase.removeChild(ContainerBase.java:936)
Please can any one help me to clear these errors.
Thanks in advance.
My diagnossis: you have a Map implementation in a thread local, and this map doesn't support the operation entrySet, which is triggered by Map#toString. To be precise, your exception is thrown from this line of code in com.sun.xml.internal.stream.writers.XMLStreamWriterImpl.
Tomcat's code that clears the thread local is quite unfortunately written to unconditionally call toString on objects just to be able to log them if the debug level is on.
If you can't get rid of using a thread-local for this, you may have quite some trouble working around this problem.
Your thread leak, by the way, is very probably the result of failed cleanup due to the above error.
Below is outtake of unique entries in my Catalina.out file on CentOS machine. I'm running Tomcat 6 with spring 3 and my application. There is whole bunch of them so I just picked some that keep repeating. This doesn't happen all the time but it happens at least once a week.
The Question is what can I do to prevent the bellow from happening?
Feb 3, 2011 2:37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [] registered the JBDC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Feb 3, 2011 2:37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [com.iteezy.shared.domain.DirEntry.data] but has failed to stop it. This is very likely to create a memory leak.
Feb 3, 2011 2:37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named
[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0] but has failed to stop it. This is very likely to create a memory leak.
Feb 3, 2011 2:37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [File Reaper] but has failed to stop it. This is very likely to create a memory leak.
Feb 3, 2011 2:37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [pool-1-thread-22] but has failed to stop it. This is very likely to create a memory leak.
37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
b application [] appears to have started a thread named
[org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
37:48 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
b application [] created a ThreadLocal with key of type [net.sf.json.AbstractJSON$1] (value [net.sf.json.AbstractJSON$1#40bbb3d6]) and a value of type [java.util.HashSet] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
When you define an external flag which a thread is suppose to poll and exit when it's set - it must be volatile. Otherwise the thread might never see the change made by other thread.
However there is already a feature like that in standard API - it is called an interrupt() method and Thread.currentThread().isInterrupted(). No need to duplicate already existing logic. See: Stopping a specific java thread.
That being said calling interrupt() on each and every thread is a bad idea as well because there is no guarantee that all threads respond to it. Examining your exceptions I noticed the following threads not being cleaned up properly:
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0 - close the C3P0 data source. Since you are using Spring simply add destroy-method="close". We're done with this thread.
File Reaper - as far as I can see this thread is created by FileCleaningTracker. You need to call FileCleaningTracker.exitWhenFinished() explicitly when shutting down your application (or when the class is no longer needed, I never used it) or let Spring to do this (see above). Chances are some 3rd party library uses it and not closing properly - this means it has a bug.
pool-1-thread-22 - this is one of the threads created by Executors utility inside ExecutorService. Make sure you call shutdown() on every such a pool in your application during shutdown.
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-2 - Quartz worker thread (the one that actually runs jobs). SchedulerFactoryBean closes the scheduler for you automatically, I think Tomcat is mistaken here, I see this error often as well. Nevertheless looks like setting SchedulerFactoryBean.waitForJobsToCompleteOnShutdown to true solves this.
com.iteezy.shared.domain.DirEntry.data - I am not sure about this one. It is either your own thread that needs to be interrupted upon shutdown or H2 database thread (?) Its stack needs to be examined to guess where does it come from.
The bottom line is: don't just kill everything that moves (actually, Tomcat does that for you after issuing this warning) but determine where the threads come from and use framework/library specific close() method to allow further cleanup. Be gentle.
Set up a Servlet to manage this in its destroy() method. The threads can check a flag to see if they must continue or not.
In your servlet, do the following in the destroy method. You will obviously need to be able to have access to a Collection<MyThread> but how you get that really depends on how your system is set up.
destroy() {
for (MyThread thread : myThreads) {
thread.stopProcessing();
}
}
Your MyThread class will have something like this:
public class MyThread {
private boolean finished = false;
#Override
public void run() {
while (!finished) {
//do something
}
}
public void stopProcessing() {
finished = true;
}
}
Shutdown your webapp properly. Don't leave these threads running.
Alternatively, don't keep redeploying the webapp.
I'm using Eclipse to develop a java web app and My IT department called and said I had over 75 open oracle connections to the development Oracle Server originating from my PC. The deployed app has been using the same singleton connection bean for a year now and I haven't had any trouble with exceeding connections there. IT even connfirmed I only had 4 open connections on the deployment server. Something is going on when I develop locally in Eclipse.
Looking at the console debug output, everything looks good on initial startup
Sep 27, 2010 9:39:08 AM org.apache.catalina.startup.Catalina start
== no ConnCache Bean instance found
== Initializing Connection Cache Data Src
== About to load the properties
== Initializing Datasource for using jdbc:oracle:thin:#hostname:port:oracle-sid
== Reading property file
== Setting InitialLimit to 3
== Properties {TimeToLiveTimeout=4, MinLimit=4, ConnectionWaitTimeout=44, MaxLimit=5, InitialLimit=3, AbandonedConnectionTimeout=281, InactivityTimeout=269}
But then I noticed the following SEVERE message in the console When Eclipse reloads the context
Sep 27, 2010 9:40:38 AM org.apache.catalina.core.StandardContext reload
INFO: Reloading this Context has started
Sep 27, 2010 9:40:38 AM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [] registered the JBDC driver [oracle.jdbc.driver.OracleDriver] but failed to unregister
it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Sep 27, 2010 9:40:38 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [Thread-9] but has failed to stop it.
This is very likely to create a memory leak.
I'm guessing that Eclipse is recreating a connection cache everytime the it reloads the context and the old connections and cache are not getting destroyed. How can I get Eclipse to terminate the cache or close all previous connections when the context is reloaded. I can post my connectionBean class if necessary.
It appears that you are leaking memory via the unclosed connections of the previous version of the application. This is quite possible if you've chosen to perform automatic publishing of your web application from Eclipse to Tomcat.
Tomcat will discard the current classloader of the web application, when it attempts to perform a reload. All objects from this GC root (the classloader), that will still be held, will not be garbage collected. Due to the presence of the singleton (I'm assuming that the singleton stores a DataSource object or is maintaining a pool of Connection objects), these Connection objects will never be treated as unreachable objects, and hence, will not be eligible for GC.
In the short run, you can do the following:
If you can help it, restart Tomcat whenever you need to publish a new version of the application.
Implement a ServletContextListener, and attempt to discard the singleton reference when the servlet (application) context is destroyed. This makes the singleton object unreachable, and hence eligible for GC. This is recommended, for the same is bound to occur in production, unless Tomcat is restarted.
By the way, this behavior, where in singletons cause memory and hence connection leakage will occur in any application server, as long as the application is redeployed time and again without restarting the server. The only way to prevent this, is to ensure that the singleton is destroyed when the application context is destroyed.