I am debugging a problem that I've had for years in a Tomcat application - a memory leak caused when restarting an application since the Webapp classloader cannot be GC'd. I've taking snapshots of the heap with JProfiler and it seems that at least some my static variables aren't getting freed up.
Certain classes have a static final member which is initialized when the class is first loaded, and because it's final I can't set it to null on app shutdown.
Are static final variables an anti-pattern in Tomcat, or am I missing something? I've just starting poking around with JProfiler 8 so I may be misinterpreting what the incoming references are telling me.
Cheers!
Luke
It is from a few years ago but this presentation I gave at JavaOne covers this topic exactly. The key steps to find the leak are in slide 11 but there is a lot of background information that might be useful as well.
The short version is:
Trigger the leak
Force GC
Use a profiler to find an instance of org.apache.catalina.loader.WebappClassLoader that has the property started=false
Trace the GC roots of that object - those are your leaks
As I note in the presentation, finding the leaks is one thing, finding what triggered them can be a lot harder.
I would recommend running on the latest stable Tomcat version as we are always improving the memory leak detection and prevention code and the warnings and errors that that generates may also provide some pointers.
Static variables should be garbage collected when the class itself is garbage collected, which in turn is the case when its class loader is garbage collected.
You can easily create memory leak by having anything that wasn't loaded by the applications classloader having a reference to any of your classes (or an instance of your classes). Look for things like callback listeners etc. that you didn't remove properly (inner/anonymous classes are easily overlooked).
A single reference to one of your classes prevents its class loader and in turn any class loaded by that class loader to be garbage collected.
Edit, example of leaking an object that prevents GC of all your classes:
MemoryMXBean mx = ManagementFactory.getMemoryMXBean();
NotificationListener nl = new NotificationListener() { ... };
((NotificationEmitter) mx).addNotificationListener(nl, ..., ...);
If you register a listener (NotificationListener here) with an object that exists ouside of your applications scope (MemoryMXBean here), your listener will stay 'live' until its explicitly removed. Since your listener instance hold a reference to its ClassLoader (your application classloader) you have now created a strong reference chain preventing GC of the classloader, and in turn, all the classes it loaded, and through that, any static variables those classes hold.
Edit2: Basically you need to avoid this situation:
[Root ClassLoader]
|
v
[Application ClassLoader]
|
v
(Type loaded by Root).addSomething()
The JVM running the application server has loaded the JRE trough the root class loader (and possibly the application server, too). That means those classes will never become eligible for GC, since there will always be live references to some of them. The application server will load your application in a separate class loader that it will not hold a reference to any longer when your application is redeployed (or at least should). But your application will share all classes from at least the JRE with the application server (at least the JRE, but commonly also the Application Server).
In the hypothetical case when the application server were to create a separate class loader (with no parent, a second root class loader practically) and try to load the JRE a second time (as private to your application) it would cause a lot of problems. Classes intended to be singletons would exists twice, and the two class hierarchies would be incapable of holding any refrences of the other (Caused by the same class loaded by different class loaders beeing different types for the JVM). They couldn't even use java.lang.Object as a reference type for the respective "other" class loaders objects.
This Blog can give you an idea about the memory leak in your application.
Related
Problem background
I run a Groovy/Grails system that dynamically compiles and loads user-defined code. This is basically done via GroovyClassLoader.
I'm seeing the dynamic classes themselves both loaded AND unloaded just fine. I have added the
-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
flags and that works fine. The dynamic classes are being unloaded when the classloader is GC'ed.
Problem description
However my heap memory is leaking, and this is the source of the leak:
Now, the Launcher$AppClassLoader has a property parallelLockMap defined in java.lang.ClassLoader. Apparently this ConcurrentHashMap is used to hold parallel class loading locks in Java 7. Check out the source code.
The entries held in this map have String keys (the class name for which the lock applies) and Object values (the lock objects). There are two kinds of leaking keys:
"groovy.runtime.metaclass.MyDynamicallyLoadedClassNameMetaClass"
"MyDynamicallyLoadedClassNameBeanInfo"
So to me it looks like these keys are created in parallelLockMap when the *MetaClass and *BeanInfo classes are loaded, but they are not removed when the associated dynamically loaded class (and its metaclasses) are unloaded. Now this is getting quite deep into the internals of Groovy and its metaclass system, and I'm running out of expertise.
Test Case
This will eventually run you out of heap space, although it will take a while:
String newClass = "class CLASSNAME {}"
while (true) {
GroovyClassLoader gcl = new GroovyClassLoader()
Class clazz = gcl.parseClass(newClass.replace("CLASSNAME", "NewClass"+System.nanoTime()))
clazz.newInstance()
}
Be sure to run this with the above JVM flags so that you run out of heap space, not PermGen space. Again, PermGen gets garbage collected nicely, no leak there.
Questions
1) Is this a bug in Groovy or Java 7? Whose responsibility would it be to clean the parallelLockMap? Should I submit an issue report?
2) Is there a workaround? I'm thinking of using a custom ClassLoader that does not first try to delegate class loading to its parent for these MetaClass and BeanInfo classes, thus preventing the call to java.lang.ClassLoader#loadClass(..). I'm not an expert in Java/Groovy classloading, though.
EDIT: I have submitted a Groovy JIRA here.
EDIT: On the JDK side this issue has recently been reported here.
I know that static field can cause a memory leak, because they will not be GCed.
But when there is an web application which is deployed in a container (such as Tomcat), each application has its own ClassLoader, and it can be undeployed.
My question is, do garbage collector claims objects referenced by static members of the classes which are going to be unloaded?
The simplest case is a singleton (implemented by an static variable referencing self), will it be GCed if the application is undeployed?
This might answer your question:
When an app is stopped, Tomcat (even before 6.0.24) nullifies the
value of all static class variables of classes loaded by the
WebAppClassLoader. In some cases, it may fix a classloader leak (for
example because of a custom ThreadLocal class, see above), but even if
we still have a leak, it may decrease the amount of memory lost
You can read more here
Cheers !!
When I redeploy my application in tomcat, I get the following issue:
The web application [] created a ThreadLocal with key of type
[java.lang.ThreadLocal] (value [java.lang.ThreadLocal#10d16b])
and a value of type [com.sun.xml.bind.v2.runtime.property.SingleElementLeafProperty]
(value [com.sun.xml.bind.v2.runtime.property.SingleElementLeafProperty#1a183d2]) but
failed to remove it when the web application was stopped.
This is very likely to create a memory leak.
Also, am using ehcache in my application. This also seems to result in the following exception.
SEVERE: The web application [] created a ThreadLocal with key of type [null]
(value [com.sun.xml.bind.v2.ClassFactory$1#24cdc7]) and a value of type [java
.util.WeakHashMap...
The ehcache seems to create a weak hash map and I get the message that this is very likely to create a memory leak.
I searched over the net and found this,
http://jira.pentaho.com/browse/PRD-3616 but I dont have access to the server as such.
Please let me know if these warnings have any functional impact or can they be ignored? I used the "Find Memory leaks" option in tomcat manager and it says "No memory leaks found"
When you redeploy your application, Tomcat creates a new class loader. The old class loader must be garbage collected, otherwise you get a permgen memory leak.
Tomcat cannot check if the garbage collection will work or not, but it knows about several common points of failures. If the webapp class loader sets a ThreadLocal with an instance whose class was loaded by the webapp class loader itself, the servlet thread holds a reference to that instance. This means that the class loader will not be garbage collected.
Tomcat does a number of such detections, see here for more information. Cleaning thread locals is difficult, you would have to call remove() on the ThreadLocal in each of the threads that is was accessed from. In practice this is only important during development when you redeploy your web app multiple times. In production, you probably do not redeploy, so this can be ignored.
To really find out which instances define the thread locals, you have to use a profiler. For example the heap walker in JProfiler (disclaimer: my company develops JProfiler) will help you to find those thread locals. Select the reported value class (com.sun.xml.bind.v2.runtime.property.SingleElementLeafProperty or com.sun.xml.bind.v2.ClassFactory) and show the cumulated incoming references. One of those will be a java.lang.ThreadLocal$ThreadLocalMap$Entry. Select the referenced objects for that incoming reference type and switch to the allocations view. You will see where the instance has been allocated. With that information you can decide whether you can do something about it or not.
Mattias Jiderhamn has an excellent
6-part article that explains very clearly the theory and practice about classloader leaks. Even better, he also released a jar file that we can include in our war files. I tried it on my web apps, and the jar file worked like a charm! The jar file is called classloader-leak-prevention.jar. To use it is as simple as just adding this to our web.xml
<listener>
<listener-class>se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor</listener-class>
</listener>
and then adding this to our pom.xml
<dependency>
<groupId>se.jiderhamn</groupId>
<artifactId>classloader-leak-prevention</artifactId>
<version>1.15.2</version>
</dependency>
For more information, please refer to the
project home page hosted on GitHub
or
Part 6 of his article
Creating Threads without cleaning them up correctly will eventually run you out of memory - been there, done that.
Those who are still wondering for quick solution/workaround, can go for below:
If running the standalone tomcat, kill javaw.exe or the process bearing it.
If running from eclipse, kill eclipse.exe and java.exe or enclosing process.
Still not resolved, Check for the task manager, it is likely that the process which is causing this will be shown with highest memory
usage - Do your analysis and kill that.
You should be good to redeploy the stuff and proceed without memory issues.
I guesses you probably seen this but just in case ehcache doc recommends to put the lib in tomcat and not in WEB-INF/lib.
I recommend initializing thread locals, in a ServletRequestListener.
ServletRequestListener has 2 methods: one for initialization and one for destruction.
This way, you can cleanup your ThreadLocal. Example:
public class ContextInitiator implements ServletRequestListener {
#Override
public void requestInitialized(ServletRequestEvent sre) {
context = new ThreadLocal<ContextThreadLocal>() {
#Override
protected ContextThreadLocal initialValue() {
ContextThreadLocal context = new ContextThreadLocal();
return context;
}
};
context.get().setRequest(sre.getServletRequest());
}
#Override
public void requestDestroyed(ServletRequestEvent sre) {
context.remove();
}
}
web.xml:
<listener>
<listener-class>ContextInitiator</listener-class>
</listener>
In Java when I have an application running in appserver like glassfish, with my application deployed as EJB. When I undeploy EJB what happens to to sigletone classes that a loaded into memory. I understand until I restart the container they are present there and can be garbage collected but I am not sure where and when it will happen, So if I deploy the ejb once again it may pick up the old objects from jvm,
?
Each deployed app is loaded with its own separate classloader. Since the classloader is part of a class's identity, the same class can be loaded multiple times (with different configuration) without the different instances interfering with each other.
This effectively isolates different applications within an app server from each other and even allows the same application to be run twice in parallel.
When an application is undeployed, all its objects (including the classloader and thus the classes themselves) will be garbage collected, if no references to them remain. Unfortunately, it can happen that references remain in some system class and prevent the garbage collection - this is called a classloader leak.
Garbage collection is run at arbitrary times and you cannot control it. While Java's garbage collector can reuse old objects, I guess this is not the case. When you undeploy, the singleton gets destroyed. When you deploy, it creates a new one.
I recently began profiling an osgi java application that I am writing using VisualVM. One thing I have noticed is that when the application starts sending data to a client (over JMS), the number of loaded classes starts increasing at a steady rate. The Heap size and the PermGen size remains constant, however. The number of classes never falls, even after it stops sending data. Is this a memory leak? I think it is, because the loaded classes have to be stored somewhere, however the heap and permgen never increase even after I run the application for several hours.
For the screenshot of my profiling application go here
Are you dynamically creating new classes on the fly somehow?
Thanks for your help. I figured out what the problem is. In one of my classes, I was using Jaxb to create an XML string. In doing this, JAXB ueses reflection to create a new class.
JAXBContext context = JAXBContext.newInstance(this.getClass());
So although the JAXBContext wasn't saying around in the heap, the classes had been loaded.
I have run my program again, and I see a normal plateau as I would expect.
I'm willing to bet that your problem is related to bytecode generation.
Many libraries use CGLib, BCEL, Javasist or Janino to generate bytecode for new classes at runtime and then load them from controlled classloader. The only way to release these classes is to release all references to the classloader.
Since the classloader is held by each class, this also means that you should not release the references to all classes as well [1]. You can catch these with a decent profiler (I use Yourkit - search for multiple classloader instances with the same retained size)
One catch is that the JVM does not unload classes by default (the reason is backwards compatibility - that people assume (wrongly) that static initializers would be executed only once. The truth is that they get executed every time a class is loaded.) To enable unloading, you should pass some use the following options:
-XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled
(tested with JDK 1.5)
Even then, excessive bytecode generation is not a good idea, so I suggest you look in your code to find the culprit and cache the generated classes. Frequent offenders are scripting languages, dynamic proxies (including the ones generated by application servers) or huge Hibernate model (in this case you can just increase your permgen).
See also:
http://blogs.oracle.com/watt/resource/jvm-options-list.html
http://blogs.oracle.com/jonthecollector/entry/presenting_the_permanent_generation
http://forums.sun.com/thread.jspa?messageID=2833028
You might find some hotspot flags to be of use in understanding this behavior like:
-XX:+TraceClassLoading
-XX:+TraceClassUnloading
This is a good reference:
http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp
Unless I misunderstand, we're looking here at loaded classes, not instances.
When your code first references a class, the JVM has the ClassLoader go out and fetch the information about the class from a .class file or the like.
I'm not sure under what conditions it would unload a class. Certainly it should never unload any class with static information.
So I would expect a pattern roughly like yours, where as your application runs it goes into areas and references new classes, so the number of loaded classes would go up and up.
However, two things seems strange to me:
Why is it so linear?
Why doesn't it plateau?
I would expect it to trend upwards, but in a wobbly line, and then taper off on the increase as the JVM has already loaded most of the classes your program references. I mean, there are a finite number of classes referenced in most applications.
Are you dynamically creating new classes on the fly somehow?
I would suggest running a simpler test app through the same debugger to get a baseline case. Then you could consider implementing your own ClassLoader that spits out some debug information, or maybe there is a tool to make it report.
You need to figure out what these classes being loaded are.
Yes, it's usually a memory leak (since we don't really deal with memory directly, it's more of a class instance leak). I've gone through this process before and usually it's some listener added to an old toolkit that didn't remove it self.
In older code, A listener relationship causes the "listener" object to remain around. I'd look at older toolkits or ones that haven't been through many revs. Any long-existing library running on a later JDK would know about reference objects which removes the requirement for "Remove Listener".
Also, call dispose on your windows if you recreate them each time. I don't think they ever go away if you don't (Actually there is also a dispose on close setting).
Don't worry about Swing or JDK listeners, they should all use references so you should be okay.
Use the Eclipse Memory Analyzer to check for duplicated classes and memory leaks. It might happen that the same class gets loaded more than once.
Regards,
Markus