The title might be a bit strong, but let me explain how I understand what happens. I guess this happened with Tomcat (and the message cited comes from Tomcat), but I'm not sure anymore.
TL;DR At the bottom there's a summary why I'm claiming that it is the web servers' fault.
I might be wrong (but without the possibility of being wrong there would be no reason to ask):
An application uses a library
the library uses a ThreadLocal
the ThreadLocal refers to an object from the library
each object refers to its ClassLoader
The webserver
pools its worker threads for efficiency
lends an arbitrary thread to an application
does nothing special (w.r.t. the thread pool) when an application stops or redeploys
If I understand it correctly, after a redeploy the old "dirty" threads continue to be reused. Their ThreadLocals refer to the old classes which refer to their ClassLoader which refer to the whole old class hierarchy. So a lot of stuff stays in the PermGen space which over time leads to an OutOfMemoryError. Is this right so far?
I'm assuming two things:
the redeploy frequency is a few time per hour
the thread creation overhead is a fraction of a millisecond
So a complete thread pool renewal upon each redeploy costs a fraction of a millisecond a few times per hour, i.e., there's a time overhead of 0.0001 * 12/3600 * 100% i.e. 0.000033%.
But instead of accepting this tiny overhead, there are countless problems. Is my calculation wrong or what am I overlooking?
As a warning we get the message
The web application ... created a ThreadLocal with key of type ... and a value of type ... but failed to remove it when the web application was stopped.
which should be better stated as
The web server ... uses a thread pool but failed to renew it after stopping (or redeploying) an application.
Or am I wrong? The time overhead is negligible even when all threads get recreated from time to time. But clearing their ThreadLocals before they are provided to the applications would suffice and be even faster.
Summary
There are some real problems (recently this one) and the user can do nothing about it. The library writers sometimes can and sometimes can not. IMHO the web servers could solve it pretty easily. The thing happens and has a cause. So I'm blaming the only one party which could do anything about it.
Proposal for what the web server should exactly do
The title of this question is more provocative than correct, but it has its point. And so does the answer by raphw. This linked question has another open bounty.
I think the web servers could solve it as follows:
ensure that each thread gets reused (or killed) sometime
store a LastCleanupTimestamp in a ThreadLocal (for new threads it's the creation time)
when re-using a thread, check if the cleanup timestamp is below some threshold (e.g., now minus some delta, e.g., 1 hour)
if so, clean all ThreadLocals and set a new LastCleanupTimestamp
This would assure that no such leak exists longer than delta plus the duration of the longest request plus the thread turnaround time. The cost would compose as follows:
checking a single ThreadLocal (i.e., some nanoseconds) per request
cleaning all ThreadLocals reflectively (i.e., some more nanoseconds once each delta per thread)
the cost from removing the data possibly useful for the application which stored them. This can't break an application as no application can assume to see a thread containing the thread locals it has set (since it can't even assume to see the thread itself anymore), but it may cost time needed to recreate the data (e.g., a cached DateFormat instance if someone still uses such a terrible thing).
It could be switched off by simply setting the thresold, if no app has been undeployed or redeployed recently.
TL;DR It's not web servers that create memory leaks. It's you.
Let me first state the problem more explicitly: ThreadLocal variables often refer to an instance of a Class that was loaded by a ClassLoader that was meant to be exclusively used by a container's application. When this application gets undeployed, the ThreadLocal reference gets orphaned. Since each instance keeps a reference to its Class and since each Class keeps a reference to its ClassLoader and since each ClassLoader keeps a reference to all classes it ever loaded, the entire class tree of the undeployed application cannot get garbage collected and the JVM instance suffers a memory leak.
Looking at this problem, you can optimize for either:
Allow as many requests per second as possible even throughout a redeploy (thus keep response time short and reuse threads from a thread pool)
Make sure that threads stay clean by discarding threads once they were used when a redeploy occurred (thus patch forgotten manual cleaning)
Most developers of web applications would argue that the first is more important since the second can be achieved by writing good code. And what would happen when a redeploy would happen concurrently to long lasting requests? You cannot shut down the old thread pool since this would interrupt running requests. (There is no globally defined maximum for how long a request cycle can take.) In the end, you would need a quite complex protocol for that and that would bring its own problems.
The ThreadLocal induced leak can however be avoided by always writing:
myThreadLocal.set( ... );
try {
// Do something here.
} finally {
myThreadLocal.remove();
}
That way, your thread will always turn out clean. (On a side note, this is almost like creating global variables: It is almost always a terrible idea. There are some web frameworks like for example Wicket that make a lot of use of this. Web frameworks like this are terrible to use when you need to do things concurrently and get very unintuitive for others to use. There is a trend away from the typical Java one thread per request model such as demonstrated with Play and Netty. Do not get stuck with this anti-pattern. Do use ThreadLocal sparingly! It is almost always a sign of bad design.)
You should further be aware that memory leaks that are induced by ThreadLocal are not always detected. Memory leaks are detected by scanning the web server's worker thread pool for ThreadLocal variables. If a ThreadLocal variable was found the variable's Class reveals its ClassLoader. If this ClassLoader or one of its parents is that of the web application that just got undeployed, the web server can safely assume a memory leak.
However, imagine that you stored some large array of Strings in a ThreadLocal variable. How can the web server assume that this array belongs to your application? The String.class was of course loaded with the JVM's bootstrap ClassLoader instance and cannot be associated with a particular web application. By removing the array, the web server might break some other application that is running in the same container. By not removing it, the web server might leak a large amount of memory. (This time, it is not a ClassLoader and its Classes that are leaked. Depending on the size of the array, this leak might however even be worse.)
And it gets worse. This time, imagine that you stored an ArrayList in your ThreadLocal variable. The ArrayList is part of the Java standard library and therefore loaded with the system ClassLoader. Again, there is no way of telling that the instance belongs to a particular web application. However, this time your ClassLoader and all its Classes will leak as well as all instances of such classes that are stored in the thread local ArrayList. This time, the web server even cannot certainly determine that a memory leak occurred when it finds that the ClassLoader was not garbage collected since garbage collection can only be recommended to a JVM (via System#gc()) but not enforced.
Renewing the thread pool is not as cheap as you might assume.
A web application cannot just go and throw away all threads in a thread pool whenever an application is undeployed. What if you stored some values in those threads? When a web application recycles a thread, it should (I am not sure if all web servers do this) find all non-leaking thread local variables and reregister them in the replaced Thread. The numbers you stated about efficiency would therefore not longer hold.
At the same time, the web server need to implement some logic that manages the replacement of all thread pool's Threads what does neither work in favor of your proposed time calculation. (You might have to deal with long lasting requests - think of running an FTP server in a servlet container -- such that this thread pool transition logic might be active for quite a long time.)
Furthermore, ThreadLocal is not the only possibility of creating a memory leak in a servlet container.
Setting a shut down hook is another example. (And it is unfortunately a common one. Here, you should manually remove the shut down hook when your application is undeployed. This problem would not be solved by discarding threads.) Shut down hooks are furthermore instances of custom subclasses of Thread that were always loaded by an application's class loader.
In general, any application that keeps a reference to an object that was loaded by a child class loader might create a memory leak. (This is generally possible via Thread#getContextClassLoader().) In the end, it is the developer's resposibility to not cause memory leaks, even in Java applications where many developer's misinterpret the automatic garbage collection as there are no memory leaks. (Think of Jochua Bloch's famous stack implementation example.)
After this general statement, I want to comment on Tomcat's memory leak protection:
Tomcat does not promise you to detect all memory leaks but covers specific types of such leaks as they are listed in their wiki. What Tomcat actually does:
Each Thread in the JVM is examined, and the internal structures of the
Thread and ThreadLocal classes are introspected to see if either the
ThreadLocal instance or the value bound to it were loaded by the
WebAppClassLoader of the application being stopped.
Some versions of Tomcat even try to compensate for the leak:
Tomcat 6.0.24 to 6.0.26 modify internal structures of the JDK
(ThreadLocalMap) to remove the reference to the ThreadLocal instance,
but this is unsafe (see #48895) so that it became optional and
disabled by default from 6.0.27. Starting with Tomcat 7.0.6, the
threads of the pool are renewed so that the leak is safely fixed.
However, you have to properly configure Tomcat to do so. The wiki entry on its memory leak protection even warns you how you can break other applications when TimerThreads are involved or how you might leak memory leaks when starting your own Threads or ThreadPoolExecutors or when using common dependencies for several web applications.
All the clean up work offered by Tomcat is a last resort! Its nothing you want to have in your production code.
Summarized: It is not Tomcat that creates a memory leak, it is your code. Some versions of Tomcat try to compensate for such leaks which are detectable if it is configured to do so. However, it is your responsibility to take care of memory leaks and you should see Tomcat's warnings as an invitation to fix your code rather than to reconfigure Tomcat to clean up your mess. If Tomcat detects memory leaks in your application, there might even be more. So take a heap and thread dump out of your application and find out where your code is leaking.
Related
I noticed something interesting.
I was told (and from what I read) that it is safe to hold request-scope variables in a ThreadLocal (let's say you don't have access to the request object and can't use request attributes)
Well, it seems to work (at least when I checked in tomcat). e.g. Even if I have 10 threads in the pool, the thread local variables seem to live only for the scope of a single request. I mean, even if I see the same thread name (let's say I have 10 in the pool, so after 10 requests I'm ought to see some repeat) each request "magically" resets all thread local variables for that thread.
Is that true?
Is every time a request thread is returned to the pool, it clears all thread local vars? How?
Later tomcat versions have a thread local protection mechanism to avoid memory leaks and especially defend against hanging class loaders on application re-deploy.
TC 6.0.24 detects it and 7.0.6 removes the thread locals, as documented here: http://wiki.apache.org/tomcat/MemoryLeakProtection
So this is not normal for threads/threadpools, but a TC feature.
Is every time a request thread is returned to the pool, it clears all
thread local vars?
It seems that Tomcat took care of house-cleaning but in general the answer is No, and this was one of the reason that was causing Out Of Memory errors in Glassfish 3.0.1
In Glassfish 3.0.1, for instance during application deployment some
code was creating a ThreadLocal variable holding a reference to some
instances which in turn seems to hold a reference to a lot of other
objects (mostly related to proxy classes generated during EJB and CDI
deployment). Glassfish doesn't seem to clean this reference after the
deployment finishes, which wouldn't be much of a problem if the thread
that deploys the application terminates.
Unfortunately the application deployment thread never dies, because it
is not created solely for the purpose of application deployment. It is
instead taken from the Glassfish thread pool and expected behaviour is
returned to the pool once it finishes the deployment task. That means
the ThreadLocal reference never gets cleaned and over time causes the
heap to overflow.
I am developing application which creates new classloader, executes some code with classes loaded using this classloader and then throws it away, starting another classloader and doing the same. I would like to reclaim all memory and kill all threads left over one use of the classloader.
My problem is that if the code loaded by this classloader is poorly written (or rather not expecting this use), it can leave some threads behind. Current problem in one such library is that there is a threadpool that cannot be shut down, stored as final static variable (shutdown() checks whether this is the 'default' threadpool stored in this variable and fails if it is so). I know how to modify final variables, but there's the danger that the variable lookup has been already optimized out.
I can't even shoot those threads using Thread.stop() (I am fine with inconsistent state of anything loaded in the dying classloader) because the threadpool would recreate this thread. I don't want to rewrite the shutdown code using reflection, and I am also reluctant to nulling all fields and hoping that it will crash. Generally, I'd like to keep the 'hacks' at minimum.
I know this is problem for application servers undeploying the applications - is there any good solution?
I am dealing with below OutOfMemory exception in WAS 6.1.
Exception in thread "UnitHoldingsPolicySummary" java.lang.OutOfMemoryError: unable to create new native thread.
I have done a lot of research on this to prevent this. After Googling, I have found that, this happens when the Native memory gets exhausted due to creation of lots of threads concurrently.
Now, after analysing the below logs, we can figure out that, inside the application, the threads are created explicitely, which I read is a very very bad practice. (Can experts please confirm this?)
07/07/14 08:50:38:165 BST] 0000142c SystemErr R Exception in thread "xxxxxx" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at com.fp.sv.controller.business.thread.xxxxxxxxxexecute(Unknown Source)
at com.fp.sv.controller.business.thread.xxxxxxxxx.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)
I am more into WAS administration and doesn't possess much knowledge on Java and thread creation in Java. Now I need to discuss this with developer, but before that I want to be 100% confirmed that my findings are correct and developers should correct the code by not explicitely creating the threads.
What all things that I need to check on application server side before blaming this on code?
On solaris, I am firing the command pmap -x 9547|grep -i stack|wc -l to check how many threads are getting created on that instance of time. I could see during the 'OutOfMemory' issue, this number is very high.
Could you please confirm whether this command is the good way to checknumber of threads currently active?
Editing the question with my latest findings
Also, when this issue happens, at the same one of the MQ queue gets piled up as WAS doesn't pick up the messages from the queue. I could see below error in the application specific logs.
Non recoverable Exception detected whilst connecting to queue manager or response queue
Underlying reason = MQJE001: Completion Code 2, Reason 2102
Can this issue related to MQ as well?Which in turn causes OutOfMemory issue?
Regards,
Rahul
There are different possibilities of implementing a threading system for a virtual machine. The two extreme forms are:
Green threads: All Java Thread instances are managed within one native OS thread. This can cause problems if a method blocks within a native invocation what makes this implementation complex. In the end, implementers need to introduce renegade threads for holding native locks to overcome such limitations.
Native threads: Each Java Thread instance is backed by a native OS thread.
For the named limitations of green threads, all modern JVM implementations, including HotSpot, choose the later implementation. This implies that the OS needs to reserve some memory for each created thread. Also, there is some runtime overhead for creating such a thread as it needs direct interaction with the underlying OS. At some point, these costs accumulate and the OS refuses the creation of new threads to prevent the stability of your overall system.
Threads should therefore be pooled for resue. Object pooling is normally considered bad practice as many programers used it to ease the JVM's garbage collector. This is not longer useful as modern garbage collectors are optimized for handling short-living objects. Today, by pooling objects you might in contrary slow down your system. However, if an object is backed by costly native resources (as a Thread), pooling is still a recommended practice. Look into the ExecutorService for a canonical way of pooling threads in Java.
In general, consider that context switches of threads are expensive. You should not create a new thread for small tasks, this will slow your application down. Rather make your application less concurrent. You only have a number of cores which can work concurrently in the first place, creating more threads than your (non-virtual) cores will not improve runtime performance. Are you implementing some sort of divide-and-conquer algorithm? Look into Java's ForkJoinPool.
Yes, it's a bad practice. Normally, you don't manage threads inside a Java EE server. By "normally" I mean "while developing business applications".
According to http://www.oracle.com/technetwork/java/restrictions-142267.html:
Why is thread creation and management disallowed?
The EJB specification assigns to the EJB container the responsibility
for managing threads. Allowing enterprise bean instances to create and
manage threads would interfere with the container's ability to control
its components' lifecycle. Thread management is not a business
function, it is an implementation detail, and is typically complicated
and platform-specific. Letting the container manage threads relieves
the enterprise bean developer of dealing with threading issues.
Multithreaded applications are still possible, but control of
multithreading is located in the container, not in the enterprise
bean.
However, I don't think your logs demonstrate that threads are being created explicitly. If you want to be 100% sure, decompile the deployables and look at the code in those lines.
Also take a look at this:
"java.lang.OutOfMemoryError : unable to create new native Thread"
And this:
https://plumbr.eu/outofmemoryerror/unable-to-create-new-native-thread
Concerning the number of threads used by your app, I'd try to use a monitoring tool like JConsole, or VisualVm.
A question came up at work today:
Imagine that I have a lot of concurrent users accessing my website, each with their own data stored in user sessions. Considering that I have a limited amount of memory available in my JVM, what happens, or what is supposed to happen, when the concurrent active sessions reach a point where the container JVM runs out of memory?
I tried to find something in the servlet spec about what should happen but there is nothing definitive in there, it just says that the developer will have access to the session objects, etc. This makes me think that it must be provider specific.
Taking Tomcat as an example, does the webserver just start throwing OutOfMemoryExceptions? Or is it more clever about the way it deals with this, for example paging the sessions off to another cache? Or another way all together?
The servlet specification does not say anything about memory considerations, so you'll get no help there.
Unless configured specifically to do otherwise, Tomcat will allow you to use-up all available memory with your HttpSession objects (realy, their contents) and ultimately the JVM will start throwing OutOfMemoryErrors, potentially taking your server down (though the JVM will continue to run, many things will behave ... unpredictably and unfortunately).
If a single request starts using a lot of memory in local variables, etc., the request-processing thread will suffer an OutOfMemoryError and stop processing the current request. (I believe in this case, the request-processing thread will actually be recycled by the Tomcat request-processing thread pool). The garbage collector will likely run shortly thereafter and re-claim the memory used by that request and your server will stabilize.
On the other hand, if you use-up a lot of memory and store those objects into the user's HttpSession, then the GC cannot free any memory and your server will continuously suffer from OutOfMemoryErrors. Though Tomcat will expire sessions on a specified schedule (the default is after 30 minutes of inactivity), the session-cleansing thread may encounter an OutOfMemoryError during operation and therefore fail to perform its duty, compounding the whole situation (because, effectively, HttpSessions will never expire).
There are several ways to mitigate the above unfortunate scenario. Which one makes sense to you is up to your requirements and environment.
Increase your heap size. This will obviously only get you so far. If you are filling-up a 16GiB heap with HttpSession and related objects, then you are reaching the limits of commodity hardware and you can't simply buy-a-bigger-box.
Reduce the session expiration time (default is 30 minutes). If a session isn't explicitly terminated (say, through a logout function), then the HttpSession object and all its contents stick around until the expiration interval has passed. If you have lots of data in lots of abandoned sessions, reducing the session expiration time may give you some breathing room.
Stop putting so much data into the user's HttpSession. While this may seem like an obnoxious suggestion ("stop doing that"), it is honestly a valid one: do you really need to store so much stuff in the session itself? How about using a data store of some kind (relational database, non-relational database like Cassandra, webcache, etc.)? Maybe store the data in a file on the disk? While these suggestions will certainly limit your ability to access your data quickly, they are certainly better options than having your server come crashing down under the weight of so much stuff in your session.
Use Tomcat's PersistentManager, which is a session manager capable of swapping active (but idle) sessions to some external storage (file-based and JDBC-based storage mechanisms are available by default). This can get you a long way while you figure out some other place to put all that data you are shoving into the user's session.
All but #4 are applicable to any servlet container. #4 should be available to JBoss users, as JBoss uses Tomcat internally. For other containers, you may find that a similar capability exists.
Generally this is vendor dependent. It is, however, frequent that containers can serialize sessions to disk at its discretion (if the session data can be serialized).
If the container can do this, it can push out inactive sessions when needed and restore them again when needed, using memory better and thereby postponing any OutOfMemoryExceptions.
I have read that it is possible with Tomcat 5.5+ to deploy a war to a Tomcat server without a restart. That sounds fantastic but I guess I am too skeptical about this functionality and it's reliability. My previous experience (with Websphere) was that it was a best practice to restart the server to avoid memory problems, etc. So I wanted to get feedback as to what pitfalls might exist with Tomcat.
(To be clear about my experience, I developed java web apps for 5 years for a large company that partitioned the app developers from the app server engineers - we used Websphere - so I don't have a lot of experience with running/configuring any app servers myself)
In general, there are multiple type of leaks and they apply to redeploy-scenarios. For production systems, it's really the best to perform restarts if possible, as there are so many different components and libraries used in todays applications that it's very hard to find them all and even harder to fix them. Esp. if you haven't got access to all source code.
Memory leaks
Thread and ThreadLocal leaks
ClassLoader leaks
System resource leaks
Connection leaks
ClassLoader leaks are the ones which bite at redeployment.
They can be caused by everything. Really, i mean everything:
Timers: Timers have Threads and Threads created at runtime inherit the current context class loader, which means the WebappClassloader of Tomcat.
ThreadLocals: ThreadLocals are bound to the thread. App servers use Thread pools. When a ThreadLocal is bound to a Thread and the Thread is given back to the pool, the ThreadLocal will stay there if nobody removes() it properly. Happens quite often and very hard to find (ThreadLocals do not have a name, except the rarely used Spring NamedThreadLocal). If the ThreadLocal holds a class loaded by the WebappClassloader, you got a ClassLoader leak.
Caches: e.g. EhCache CacheManager
Reflection: JavaBeans Introspector (e.g. holding Class or Method caches)
JDBC Drivers: they shouldn't be in the .war file anyway. Leak due to static registry
Static libraries which cache ClassLoaders, such as Commons-Logging LogFactory
Specific to Tomcat, my experience is as follows:
For simple apps with "clean" libraries, it works fine in Tomcat
Tomcat tries very hard to clean up classes loaded by the WebappClassloader. For example, all static fields of classes are set to null when a webapp is undeployed. This sometimes leads to NullPointerExceptions when code is run while the undeployment is happening, e.g. background jobs using a Logger
Tomcat has a Listener which cleans up even more stuff. Its called org.apache.catalina.core.JreMemoryLeakPreventionListener and was submitted recently to Tomcat 6.x
I wrote a blog post about my experience with leaks when doing redeployment stresstesting - trying to "fix" all possible leaks of an enterprise-grade Java Web Application.
Hot deployment is very nice as it usually is much faster than bringing the server up and down.
mhaller has written a lot about avoiding leaks. Another issue is for active users to have their session survive the application "reboot". There are several things that must be taken care of, but which all in all means that their session must be serializable and THEN deserialize properly afterwards. This can be a bit tricky if you have stateful database connections etc, but if your code is robust against database hickups anyway that shouldn't be too bad.
Also note that some IDE's allow updating code inside the WAR (in the same way as applications) when saving a modified source file, instead of having to redeploy. MyEclipse does this rather nicely.