Develop Reference

GC gets triggered often

I would like to understand why the GC gets triggered even though I have plenty of heap left unused.. I have allocated 1.7 GB of RAM. I still see 10% of GC CPU usage often.
I use this - -XX:+UseG1GC with Java 17

JVMs will always have some gc threads running (unless you use Epsilon GC which perform no gc, I do not recommend using this unless you know why you need it), because the JVM manages memory for you.
Heap in G1 is divided two spaces: young and old. All objects are created in young space. When the young space fills (it always do eventually, unless you are developing zero garbage), it will trigger some gc cleaning unreferenced objects from the young and promoting some objects which are still referenced to old.
Those spikes in the right screenshot will correspond to young collection events (where unreferenced objects get cleaned). Young space is always much more small than the old space. So it fills frequently. That is why you see those spikes regarding there is much more memory free.
DISCLAIMER This is a really very high level explanation of memory management in the JVM. Some important concepts have been not mentioned.
You can read more about g1 gc collector here
Also take a look at jstat tool which will help you understand what is happening in your heap.

Where is my JVM memory leak? Garbage collector is working properly?

I have a Scala daemon application, that runs in a server in Rackspace with a limit of 2GB. Because of an unknown reason, the server get stuck after some time the application is running. I am suspecting there is a memory leak, because the server memory gets full after some time.
I tried to run jvisualvm, making snapshots of memory in two different moments and compare them to see if there were objects that remained allocated, but I could not find anything.
The heap allocation is just around 400MB. Here is a snapshot of the JVM memory in New Relic:
Notice that PS Eden Space heap is what keeps increasing. I did a work around that kills the application every 3 hours and starts it back again (this is why the graph suddenly goes back down).
Any idea of why this PS Eden is increasing? How to fix it?
Edit 1:
Screenshot of the machine that halted minutes before 13:00
Edit 2:
On a new round, a left the server hang itself, and used G1GC. Here is the new relic graph for this run:

It's normal that the Eden grows constantly, that is where new objects are allocated. Eden will keep growing until it get's full or until a partial collection runs that collects unused objects, and shifts objects being used to the survivor region S0.
This is as per the way this type of garbage collection was designed. The idea is that it's OK that Eden is full, we let it grow and garbage collect it only when it's most convenient, minimizing the impact for application code.
Try to remove the workaround, let the server freeze and see if there are any out of memory errors in the logs. Too many classes would have cause such errors.
Try to see if the OldGen is full. Then using visualvm, force a garbage collection, and see if it goes down. If it doesn't, there is the problem.
Then take a heap dump and a thread dump and analyse the heap dump in MAT - Eclipse Memory Analyser tool, see this tutorial as well. it could be that the server just needs more memory.
One important notion, in Java there is really no notion of a memory leak, the garbage collector works mostly flawlessly to collect unused objects.
Usually the problem comes from objects that are created but are for example kept around in static collections or thread local variables accidentally, and because they are referenced get never collected.
A tool that has a free trial and allows to generate a report that pinpoints a lot of these common causes is Plumbr. That is probably the best chance at a quick solution, try to run plumbr to see if it finds something, if not then MAT analisys of the heap dump.

What causes the JVM to do a major garbage collection?

I have a Java app which shows different GC behaviors in different environments. In one environment, the heap usage graph is a slow sawtooth with major GCs every 10 hours or so, only when the heap is >90% full. In another environment, the JVM does major GCs every hour on the dot (the heap is normally between 10% and 30% at these times).
My question is, what are the factors which cause the JVM to decide to do a major GC?
Obviously it collects when the heap is nearly full, but there is some other cause at play which I am guessing is related to an hourly scheduled task within my app (although there is no spike in memory usage at this time).
I assume GC behaviour depends heavily on the JVM; I am using:
Java HotSpot(TM) 64-Bit Server VM 1.7.0_21 Oracle Corporation
No specific GC options, so using the default settings for 64-bit server (PS MarkSweep and PS Scavenge)
Other info:
This is a web app running in Tomcat 6.
Perm gen hovers around 10% in both environments.
The environment with the sawtooth behaviour has 7Gb max heap, the other has 14Gb.
Please, no guesswork. The JVM must have rules for deciding when to perform a major GC, and these rules must be encoded deep in the source somewhere. If anyone knows what they are, or where they are documented, please share!

I have found four conditions that can cause a major GC (given my JVM config):
The old gen area is full (even if it can be grown, a major GC will still be run first)
The perm gen area is full (even if it can be grown, a major GC will still be run first)
Someone is manually calling System.gc(): a bad library or something related to RMI (see links 1, 2 and 3)
The young gen areas are all full and nothing is ready to be moved into old gen (see 1)
As others have commented, cases 1 and 2 can be improved by allocating plenty of heap and permgen, and setting -Xms and -Xmx to the same value (along with the perm equivalents) to avoid dynamic heap resizing.
Case 3 can be avoided using the -XX:+DisableExplicitGC flag.
Case 4 requires more involved tuning, e.g., -XX:NewRatio=N (see Oracle's tuning guide).

Garbage collection is a pretty complicated topic, and while you could learn all the details about this, I think what’s happening in your case is pretty simple.
Sun’s Garbage Collection Tuning guide, under the “Explicit Garbage Collection” heading, warns:
applications can interact with garbage collection … by invoking full garbage collections explicitly … This can force a major collection to be done when it may not be necessary … One of the most commonly encountered uses of explicit garbage collection occurs with RMI … RMI forces full collections periodically
That guide says that the default time between garbage collections is one minute, but the sun.rmi Properties reference, under sun.rmi.dgc.server.gcInterval says:
The default value is 3600000 milliseconds (one hour).
If you’re seeing major collections every hour in one application but not another, it’s probably because the application is using RMI, possibly only internally, and you haven’t added -XX:+DisableExplicitGC to the startup flags.
Disable explicit GC, or test this hypothesis by setting -Dsun.rmi.dgc.server.gcInterval=7200000 and observing if GCs happen every two hours instead.

It depends on your configurations, since HotSpot configures itself differently in different Java environments. For example, in a server with more than 2GB and two processors some JVMs will be configured in '-server' mode instead of the default '-client' mode, which configure the sizes of the memory spaces (generations) differently, and that has an impact as to when garbage collection will occur.
A full GC can occur automatically, but also if you call the garbage collector in your code (ex: using System.gc()). Automatically, it depends on how the minor collections are behaving.
There are at least two algorithms being used. If you are using defaults, a copying algorithm is used for minor collections, and a mark-sweep algorithm for major collections.
A copying algorithm consists of copying used memory from one block to another, and then clearing the space containing the blocks with no references to them. The copying algorithm in the JVM uses uses a large area for objects that are created for the first time (called Eden), and two smaller ones (called survivors). Surviving objects are copied once from Eden and several times from the survivor spaces during each minor collection until they become tenured and are copied to another space (called tenured space) where they can only be removed in a major collection.
Most of the objects in Eden die quickly, so the first collection copies the surviving objects to the survivor spaces (which are by default much smaller). There are two survivors s1 and s2. Every time the Eden fills, the surviving objects from Eden and s1 are copied to s2, Eden and s1 are cleared. Next time, survivors from Eden and s2 are copied back to s1. They keep on being copied from s1 to s2 to s1 until a certain number of copies is reached, or because a block is too big and doesn't fit, or some other criteria. Then the surviving memory block is copied to the tenured generation.
The tenured objects are not affected by the minor collections. They accumulate until the area gets full (or the garbage collector is called). Then the JVM will run a mark-sweep algorithm in a major collection which will preserve only the surviving objects that still have references.
If you have larger objects that don't fit into the survivors, they might be copied directly to the tenured space, which will fill more quickly and you will get major collections more frequently.
Also, the sizes of the survivor spaces, amount of copies between s1 and s2, Eden size related to the size of s1 and s2, size of the tenured generation, all these may be automatically configured differently in different environments with JVM ergonomics, which may automatically select a -server or -client behavior. You might try to run both JVMs as -server or -client and check if they still behave differently.

Even if this will get down votes... My best guess (you will have to test this) would be that the heap needs to expand and when this happens a full gc will be triggered. Not all memory is allocated at once to JVM.
You can test this by setting -Xms and -Xmx to the same value, for example 7GB each

Identify old gen in heap dump (or take heap dump of old gen only)

I think I have a memory leak.
(they say the first step is admitting the problem, right?)
Anyway, I think I do - see attached image for heap by regions: .
Green is Eden, blue/red is S0/S1, purple is old. I have unlimited tenuring (>15), lots of time passed between memory being allocated and it spilling to old gen. Hence - a memory leak. I think.
So - the question - how can I analyze what is leaking? As you can see, my Eden is very active. Lot's of objects being created and destroyed all the time.
Is there a way of taking a heap dump of the old gen only? Or somehow identify the old gen in a full heap dump (if so, with what tool)?
Edit 1:
Clarification: I'm not doing anything that should retain objects in memory. Everything I allocate after the initial startup should die young.
Edit2:
New findings: I took a heap dump, GCed like crazy and took another. The second one shows a significantly reduced level of old gen usage. The main difference between the two were objects held by finalizers.
Don't finalizers run in young GC cycles? Do they always wait for a full GC to be cleaned?

seeing some things propagate to old gen isn't a huge concern. After your old gen reaches a certain threshold a full GC will kick off. If that isn't able to reclaim the memory then you have an issue. The fact that you are seeing some memory allocated during a young collection shouldn't be an alarming concern.
lots of time passed between memory being allocated and it spilling to
old gen. Hence - a memory leak. I think
Not really.. just because memory is being added to old gen doesn't mean it is a memory leak. It is normal practice during a young collection that older objects get promoted to old gen. It is during those young collections when older objects get added to the old gen. This may just be your application still ramping up. In large scale applications there may be features not used every day, which may be getting into memory later then you expected.
That being said, if you really are concerned with any memory being added to the old gen and want to investigate further, I would recommend running this application on a demo environment. Attach a profiler (VisualVM will work) and load test (JMeter is good and free) your application. If you look at the objects you can get an idea of what generation an object is. You also want to see what happens when your old gen reaches a threshold where a full GC will kick off (normally in the 70%-90% range). If your old gen recovers back to the 20% threshold, then there is no leak. In some cases the old gen may never reach the point where a full GC gets kicked off, but instead level off as you expected. The load test will help identify that.
If it doesn't recover and you confirm you have a memory leak then you will want to capture a heap dump (hprof) and use a tool like MAT (Memory Analyzer Tool) to analyze the dump to find the culprit.

Using JVisualVM (part of the JDK since Java 6 Build 10 or something like that), you can look at the TYPE of objects that are in memory. That will help you track down where the leak is. Of course, it takes a lot of digging into the code, but that's the best tool I've used that always available and reliable.
Watch out for objects being passed around, it could be that you have a handle that's being kept in a list or array that's not being cleared out. I find that if I watch the number of objects being created, and kept, in JVisualVM over a period of a few minutes, I usually get an idea of where in the code to go dig for the offending objects not being released.

Why is PermGen space growing?

I've read a few articles, and I understood the following (please correct me and/or edit the question if I'm wrong):
The java heap is segmented like this:
Young Generation: objects that are created go here, this part is frequently and inexpensively garbage collected
Old Generation: objects that survive the garbage collections of the Young generation go here, this area is garbage collected less frequently and using a more CPU demanding process/algorithm (I believe it's called mark-sweep)
Edit: as stated by another user, PermGen is not a part of the region called heap
PermGen: this area is filled of your app classes metadata and many other things that do not depend on the application usage.
So, knowing this... why does my PermGen space grows when the app is under heavy load? For what I said before this space should not incrementally fill in spite of the app load, but as I said in the beginning probably I'm wrong about some assumptions.
In fact if the PermGen space is growing, is there a way of garbage collect or reset it?

Actually, in Sun's JVM Permanent Generation (PermGen) is completely separate from the heap. Are you sure you aren't looking at the Tenured Generation? It would be suspicious indeed if your Permanent Generation kept growing.
If your perm gen IS growing constantly, it is a difficult area to dig into. Generally it should grow when new classes are loaded for the first time (and potentially certain uses of reflection could also cause this). Interned strings are also stored in perm gen.
If you happen to be on Solaris, you could use jmap -permstat to dump out perm gen statistics, but that option does not appear to be available on Windows (and potentially other platforms). Here is the documentation on jmap for Java 6
From Sun's guide on JConsole (which will let you view the size of these pools):
For the HotSpot Java VM, the memory
pools for serial garbage collection
are the following.
Eden Space (heap): The pool from which memory is initially allocated
for most objects.
Survivor Space (heap): The pool containing objects that have survived
the garbage collection of the Eden
space.
Tenured Generation (heap): The pool containing objects that have existed
for some time in the survivor space.
Permanent Generation (non-heap): The pool containing all the reflective
data of the virtual machine itself,
such as class and method objects. With
Java VMs that use class data sharing,
this generation is divided into
read-only and read-write areas.
Code Cache (non-heap): The HotSpot Java VM also includes a code cache,
containing memory that is used for
compilation and storage of native
code.

The most common causes I've seen are:
Custom classloaders that don't carefully free up older classes after loading new ones.
Classes remaining in PermGen after redeploying an application multiple times (more common in Dev than Prod)
Heavy use of Proxy classes, which are created synthetically during runtime. It's easy to create new Proxy classes when an a single class definition could be reused for multiple instances.

This is one of the more annoying problems to debug. There are a lot of reasons you could be seeing growing permgen use. Here are 2 links I found very useful in both understanding how leaks happen as well as tracking down what is causing them.
http://frankkieviet.blogspot.com/2006/10/how-to-fix-dreaded-permgen-space.html
http://frankkieviet.blogspot.com/2006/10/classloader-leaks-dreaded-permgen-space.html

Are you doing something funky with the classloader chain? Are you calling intern() on a bunch of strings?

If you are working with Java EE application it's probably a classloader leak.
you might find the following additional links to be useful:
http://www.zeroturnaround.com/blog/rjc201/
http://www.ibm.com/developerworks/java/library/j-dclp3/index.html

The most common causes I've seen are:
Java classes are loaded
JAXBContext.newInstance
String.intern()

This is a very common problem when you are manipulating the classloader. This is seen a lot in Java EE apps when you are redeploying hibernate/cglib. For more info check out
http://opensource.atlassian.com/confluence/spring/display/DISC/Memory+leak+-+classloader+won%27t+let+go