We have an application that generates thousands of Java ClassLoaders and classes, before letting them be garbage collected. We are pretty sure there is no class loader leak, but we are getting permgen errors (and even if there is, this question is orthogonal to any potential leak).
Is there any way of specifying that certain ClassLoaders and their loaded classes are placed in the heap rather than in perm gen?
We are using the Oracle JDK Java 6 on Linux.
Edit: Looks like Java 8 will no longer have PermGen. It will be replaced by Metaspace.
http://java.dzone.com/articles/java-8-permgen-metaspace
Of course you're getting perm gen errors - that's where class loaders put classes. What would you expect?
You can map them into byte buffers, like Java NIO does it, but they don't go on the heap.
You ought to try increasing your perm gen size before taking extraordinary measures. How much perm gen space do you need? Profile it using Visual VM and see.
Is there any way of specifying that certain ClassLoaders and their loaded classes are placed in the heap rather than in perm gen?
AFAIK, there is no way to do that. AFAIK, the permgen allocation happens deep in the JVM runtime where you can't get at it. (I don't think that the classloader itself is in permgen. I think it is just certain of the JVM's internal data structures that represent the classes and their code.)
If your application really has to work that way, I think you have no choice but to make permgen large enough. (Obviously, you should also check that the real problem isn't a permgen leak.)
However, it strikes me that your application's architecture is rather strange if it needs to generate lots of classes and class loaders. I'd look to see if what it is doing couldn't be done another way.
Related
Similar to java.lang.OutOfMemoryError ( PermGen space) and java.lang.ClassNotFoundException at the opening the jsp page but with slightly more modern problems.
I have a legacy app which has a lot of JSPs. Some of them use ... to do the spaces (as a CSS cleanup on the page wasn't done yet).
Recently we've been getting out of memory errors after the system has been running for a while. I was evaluating the heap space and it seems under control, forcing periodic GCs show that it is less than 100MBs of heap, but the non-heap size is increasing. I have capped the size to 300MBs which seems like a lot already as the system generally runs at 512MB in Docker.
Anyway, I use JSPC and when I totalled the amount of bytes all the generated class files take up I get 21,981,012 bytes. which I take it goes into metaspace based on other answers I have seen. Of course that number is likely increased by a large portion as it is unpacked into memory.
So my question is, is there a way to configure Java or Tomcat to release class data from metaspace when it is not in use and reload it later if needed?
UPDATE: I explicitly limited the metaspace size in my CATALINA_OPTS so that it will fail faster -XX:MaxMetaspaceSize=200m -XX:CompressedClassSpaceSize=100m
Here is the VisualVM. The classes do not seem to get unloaded and the metaspace does not go down either.
And here is the Metaspace.
JVM, by default, handles the loading and unloading of the classes. What is your application command line? Are you seeing OOM: Metaspace or JavaHeap?
I use tomcat7. My web-app contains:
hibernate
log4j
jdom
upload jar
mysql driver
...
the total of lib is 30 jar files. My hoster says: "A lot of memory is allocated to the your program(for Perm gen and heap). Please reduce the amount of memory consumed".
How do I reduce the memory requirements of my program?
How can the reduce the perm gen memory?
please help me.
You should probably attach a memory profiler first and see what type of memory is being used and why. The answers to this question list a bunch of profilers you could try.
Heap is all the memory you allocate for storing objects in Java. Every time you use "new" you allocate on the heap and the memory will become garbage collected some time after the last reference to it is cleared. Here are some typical things you might do to reduce heap usage: Switch from an in-memory parsing API to a streaming one, perhaps by using StAX or SAX instead of JDOM. Fix memory leaks or simply longer-than-necessary object retention by cleaning up in a finally block as soon as the object is no longer needed. Optimize SQL queries to make them return only exactly the necessary data to the app, perhaps by adding a max row constraint or a more restrictive where clause. Limit the size of uploads and stream them to temp files on disk instead of keeping them in memory.
Perm gen is "permanent generation". Generally, memory in the perm gen is never freed until the JVM exits. It is mostly used to store loaded classes. So anything you can do to load fewer classes helps here. Here are some ways to load fewer classes: Try not to use multiple versions of the same jars. Try to load jars in the widest context possible, preferring Tomcat's lib directory or a shared lib directory over WEB-INF/lib. Minimize reloading of the webapp because every time you reload a webapp in Tomcat, it doesn't unload the old versions of all classes it simply loads the new ones.
I'm running a GWT+Hibernate app on Glassfish 3.1. After a few hours, I run out of Permgen space. This is without any webapp reloads. I'm running with –XX:MaxPermSize=256m –XmX1024m.
I took the advice from this page, and found that I'm leaking tons of classes- all of my Hibernate models and all of my GWT RequestFactory proxies.
The guide referenced above says to "inspect the chains, locate the accidental reference, and fix the code". Easier said than done.
The classloader always points back to an instance of org.glassfish.web.loader.WebappClassLoader. Digging further, I find lots of references from $Proxy135 and similar-named objects. But I don't know how else to follow through.
new class objects get placed into the PermGen and thus occupy an ever increasing amount of space. Regardless of how large you make the PermGen space, it will inevitably top out after enough deployments. What you need to do is take measures to flush the PermGen so that you can stabilize its size. There are two JVM flags which handle this cleaning:
-XX:+CMSPermGenSweepingEnabled
This setting includes the PermGen in a garbage collection run. By default, the PermGen space is never included in garbage collection (and thus grows without bounds).
-XX:+CMSClassUnloadingEnabled
This setting tells the PermGen garbage collection sweep to take action on class objects. By default, class objects get an exemption, even when the PermGen space is being visited during a garabage collection.
There are some OK tools to help with this, though you'd never know it. The JDK (1.6 u1 and above) ships with jhat and jmap. These tools will help significantly, especially if you use the jhat JavaScript query support.
See:
http://blog.ringerc.id.au/2011/06/java-ee-application-servers-learning.html
http://blogs.oracle.com/fkieviet/entry/classloader_leaks_the_dreaded_java
http://www.mhaller.de/archives/140-Memory-leaks-et-alii.html
http://blogs.oracle.com/sundararajan/entry/jhat_s_javascript_interface
I "solved" this by moving to Tomcat.
(I can't view the link you provided as it's blocked by websense so if I'm restating anything I apologize)
It sounds like you have a class loader leak. These are difficult to track down, add these options to the JVM Options in your instance configuration
-XX:+PrintGCDetails
-XX:+TraceClassUnloading
-XX:+TraceClassLoading
Now when you run your app, you can look at the jvm.log located in your domain/logs folder and see what's loading and unloading. Mostly likely, you'll see the same class(es) loading over and over again.
A good culprit is JAXB, especially if you're creating a new JAXBContext over and over again.
I've read a few articles, and I understood the following (please correct me and/or edit the question if I'm wrong):
The java heap is segmented like this:
Young Generation: objects that are created go here, this part is frequently and inexpensively garbage collected
Old Generation: objects that survive the garbage collections of the Young generation go here, this area is garbage collected less frequently and using a more CPU demanding process/algorithm (I believe it's called mark-sweep)
Edit: as stated by another user, PermGen is not a part of the region called heap
PermGen: this area is filled of your app classes metadata and many other things that do not depend on the application usage.
So, knowing this... why does my PermGen space grows when the app is under heavy load? For what I said before this space should not incrementally fill in spite of the app load, but as I said in the beginning probably I'm wrong about some assumptions.
In fact if the PermGen space is growing, is there a way of garbage collect or reset it?
Actually, in Sun's JVM Permanent Generation (PermGen) is completely separate from the heap. Are you sure you aren't looking at the Tenured Generation? It would be suspicious indeed if your Permanent Generation kept growing.
If your perm gen IS growing constantly, it is a difficult area to dig into. Generally it should grow when new classes are loaded for the first time (and potentially certain uses of reflection could also cause this). Interned strings are also stored in perm gen.
If you happen to be on Solaris, you could use jmap -permstat to dump out perm gen statistics, but that option does not appear to be available on Windows (and potentially other platforms). Here is the documentation on jmap for Java 6
From Sun's guide on JConsole (which will let you view the size of these pools):
For the HotSpot Java VM, the memory
pools for serial garbage collection
are the following.
Eden Space (heap): The pool from which memory is initially allocated
for most objects.
Survivor Space (heap): The pool containing objects that have survived
the garbage collection of the Eden
space.
Tenured Generation (heap): The pool containing objects that have existed
for some time in the survivor space.
Permanent Generation (non-heap): The pool containing all the reflective
data of the virtual machine itself,
such as class and method objects. With
Java VMs that use class data sharing,
this generation is divided into
read-only and read-write areas.
Code Cache (non-heap): The HotSpot Java VM also includes a code cache,
containing memory that is used for
compilation and storage of native
code.
The most common causes I've seen are:
Custom classloaders that don't carefully free up older classes after loading new ones.
Classes remaining in PermGen after redeploying an application multiple times (more common in Dev than Prod)
Heavy use of Proxy classes, which are created synthetically during runtime. It's easy to create new Proxy classes when an a single class definition could be reused for multiple instances.
This is one of the more annoying problems to debug. There are a lot of reasons you could be seeing growing permgen use. Here are 2 links I found very useful in both understanding how leaks happen as well as tracking down what is causing them.
http://frankkieviet.blogspot.com/2006/10/how-to-fix-dreaded-permgen-space.html
http://frankkieviet.blogspot.com/2006/10/classloader-leaks-dreaded-permgen-space.html
Are you doing something funky with the classloader chain? Are you calling intern() on a bunch of strings?
If you are working with Java EE application it's probably a classloader leak.
you might find the following additional links to be useful:
http://www.zeroturnaround.com/blog/rjc201/
http://www.ibm.com/developerworks/java/library/j-dclp3/index.html
The most common causes I've seen are:
Java classes are loaded
JAXBContext.newInstance
String.intern()
This is a very common problem when you are manipulating the classloader. This is seen a lot in Java EE apps when you are redeploying hibernate/cglib. For more info check out
http://opensource.atlassian.com/confluence/spring/display/DISC/Memory+leak+-+classloader+won%27t+let+go
I'm running Tomcat6 in Sun's JRE6 and every couple deploys I get OutOfMemoryException: PermGen. I've done the Googling of PermGen solutions and tried many fixes. None work. I read a lot of good things about Oracle's JRockit and how its PermGen allocation can be gigs in size (compare to Sun's 128M) and while it doesn't solve the problem, it would allow me to redeploy 100 times between PermGen exceptions compared to 2 times now.
The problem with JRockit is to use it in production you need to buy WebLogic which costs thousands of dollars. What other (free) options exist that are more forgiving of PermGen expansion? How do the below JVMs do in this area?
IBM JVM
Open JDK
Blackdown
Kaffe
...others?
Update: Some people have asked why I thought PermGen max was 128M. The reason is because any time I try to raise it above 128M my JVM fails to initialize:
[2009-06-18 01:39:44] [info] Error occurred during initialization of VM
[2009-06-18 01:39:44] [info] Could not reserve enough space for object heap
[2009-06-18 01:39:44] [395 javajni.c] [error] CreateJavaVM Failed
It's strange that it fails trying to reserve space for the object heap, though I'm not sure it's "the" heap instead of "a" heap.
I boot the JVM with 1024MB initial and 1536MB max heap.
I will close this question since it has been answered, ie. "switching is useless" and ask instead Why does my Sun JVM fail with larger PermGen settings?
I agree with Michael Borgwardt in that you can increase the PermGen size, I disagree that it's primarily due to memory leaks. PermGen space gets eaten up aggressively by applications which implement heavy use of Reflection. So basically if you have a Spring/Hibernate application running in Tomcat, be prepared to bump that PermGen space up a lot.
What gave you the idea that Sun's JVM is restricted to 128M PermGen? You can set it freely with the -XX:MaxPermSize command line option; the default is 64M.
However, the real cause of your problem is probably a memory leak in your application that prevents the classes from getting garbage collected; these can be very subtle, especially when ClassLoaders are involved, since all it takes is a single reference to any of the classes, anywhere. This article describes the problem in detail, and this one suggests ways to fix it.
Technically, the "PermGen" memory pool is a Sun JVM thing. Other JVMs don't call it that, but they all have the idea of one or more non-heap memory pools.
But if you have a problem with permgen in your Sun JVM, moving to another JVM is very unlikely solve anything, it'll just manifest itself under a different name.
If multiple redeployments are causing your problems, just boost the VM's PermGen up to large values. We tried JRockit a while back because of this very problem, and it suffers from the same redeployment exhaustion. We moved back to SUn JVM.
Changing JVM is not a panacea. You can get new unexpected issues (e.g. see an article about launching an application under 4 different JVM).
You can have a class leak (e.g. via classloaders) that mostly often happen on redeploy. Frankly, I've never saw working hot redeploy on Tomcat (hope to see one day).
You can have incorrect JVM paramaters (e.g. for Sun JDK 6 64 bits -XX:+UseParNewGC switch leads to leak PermGen segment of memory. If you add additional switches: -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled-XX:+CMSPermGenSweepingEnabled the situation will be resolved. Funny, but I never met above mentioned leak with Sun JDK 6 32 bits). Link to an article "Tuning JVM Garbage Collection for Production Deployments".
Your PermGen chunk can be not enough to load classes and related information (actually that most often happens after redeploy under Tomcat, old classes stay in memory and new ones are loading)
From my past experience, debugging that kind of leak is one of the most tricky kind of debugging that I've ever had.
[UPDATED]
Useful article how to eliminate classloader link on an application redeploy.
I use JRockit and I still get PermGen errors if I don't bump up (via -XX:MaxPermSize) the memory. I also can't get anything to work to avoid getting this (other than increasing it).
Perm gen is probably the simplest memory to handle, I doubt there'd be much difference between the various vm implementations.
Make sure all those Tomcat configs that are marked turn off in production are turned off in production.
Yes, some frameworks that do generate a lot of classes an the fly, but they should be cleaning up after themselves, and, in any case, you can fit more than a few classes in 128Mb.
Seriously, if perm gen keeps going up then thats a leak a should be fixed, though it may not be your problem to fix.
The IBM JVM does not (and did not in 2009) have a permgen. You can read more about its Generational Concurrent Garbage Collector which is its default GC for Java 7.
I have sometimes run the Eclipse IDE on IBM JVM specifically because with my favorite plugins it would frequently fill up the HotSpot JVM's permgen. Sure, there was probably a memory leak that someone should have fixed, but meanwhile my IDE was not crashing and I was not busy experimenting with different settings.