This question is limited in scope to HotSpot generations. Is there any way to programmatically find out in which generation a particular instance lives. Data such as:
Young or old generation?
If young, which survivor space?
Inside TLAB? Which thread?
Any technique (ex., BTrace, JVMTI) works so long as I can do something like this:
Object x = new Object();
HotSpotGenerationInfo info = HotSpotGenerationUtil.getInfo(x);
Beggars can't be choosers but ideally I could also learn when the instance of interest was being moved from one generation to another at the moment it happens (i.e., event callback based -- not interested in the delay & overhead implicit in polling.)
Not interested in answers that just say "no" without justification :)
As far as I know, you can not directly query which memory pool an object currently lives in. However, objects are promoted to a different memory pool by a garbage collection run, and you can query the number of major/minor gc runs since VM start using JMX. If you additionally take note of these counters when the object is created, you can reconstruct whether there was a GC since and from that which pool the object lives in.
There's an additional complication to the "count the number of GCs since the object was created" approach - it doesn't take into account premature object promotion.
If the survivor spaces are basically too small, and memory pressure from Eden (ie the rate of objects surviving at least once) is high, then objects will be promoted to tenured before they hit the full tenuring threshold.
In real examples, healthy applications will typically have non-zero percentages of premature promotion. In fact, a 0% premature promotion rate is a really bad sign - it says that your survivor spaces are much, much too big and you're wasting a lot of memory.
Related
I'm trying to understand what the -XX:G1ReservePercent actually does. The descriptions I found in the official documentation are not really comprehensive:
Sets the percentage of reserve memory to keep free so as to reduce the
risk of to-space overflows. The default is 10 percent. When you
increase or decrease the percentage, make sure to adjust the total
Java heap by the same amount.
and the desciption of to-space exhausted log entry is this:
When you see to-space overflow/exhausted messages in your logs, the G1
GC does not have enough memory for either survivor or promoted
objects, or for both.
[...]
To alleviate the problem, try the following adjustments:
Increase the value of the -XX:G1ReservePercent option (and the total
heap accordingly) to increase the amount of reserve memory for "to-space".
[...]
Judging by the quote the to-space exhausted means that when performing mixed evacuation we do not have enough a free region to move survivors to.
But this then contradicts to the following official tuning advice in case of Full GC (emphasize mine):
Force G1 to start marking earlier. G1 automatically determines the
Initiating Heap Occupancy Percent (IHOP) threshold based on earlier
application behavior. If the application behavior changes, these
predictions might be wrong. There are two options: Lower the target
occupancy for when to start space-reclamation by increasing the buffer
used in an adaptive IHOP calculation by modifying
-XX:G1ReservePercent;
So what is the buffer and what does setting -XX:G1ReservePercent do (From the first glance AdaptiveIHOP has nothing to do with it...) ?
Does it keep some heap space always reserved so when mixed evacuation occur we always have free regiong to move survivors to?
Or the space is used for G1 internal housekeeping tasks? If so it is not clear what data the to-space contains so it exhausted?
To me, understanding what it really does, means to go the source code. I chose jdk-15.
The best description of this flag is here:
It determines the minimum reserve we should have in the heap to minimize the probability of promotion failure.
Excellent, so this has to do with "promotion failures" (whatever those are). But according to you quote in bold, this has something to do with AdaptiveIHOP also? Well yes, this parameter matters only in AdaptiveIHOP (which is on by default).
Another thing to notice is that G1ReservePercentage has no guarantee to be maintained all the time, it is a best effort.
If you look at how it is used in the very next line:
if (_free_regions_at_end_of_collection > _reserve_regions) {
absolute_max_length = _free_regions_at_end_of_collection - _reserve_regions;
}
things start to make some sense (for that bold statement). Notice how _reserve_regions are extracted for some computation. G1 will reserve that space for "promotion failures" (we will get to that).
If G1 reserves that space, it means less space is available for actual allocations. So if you "increase this buffer" (you make G1ReservePercent bigger as the quote suggests), your space for new objects becomes smaller and as such the time when GC needs to kick in will come sooner, so the time for when space reclamation needs to happen will come sooner too ("Lower the target occupancy for when to start space-reclamation..."). It is one complicated sentence, but in simple words it means that :
If you increase G1ReservePercentage, space will be needed to be reclaimed faster (more often GC calls).
Which, to be fair, is obvious. Not that I agree that you should increase that value, but this is what that sentence says.
After a certain GC cycle, be that minor, mixed or major, G1 knows how many regions are free:
_free_regions_at_end_of_collection = _g1h->num_free_regions();
Which of course, is the length of "free list":
return _free_list.length();
Based on this value and _reserve_regions (G1ReservePercent) plus the target pause time (200 ms by default) it can compute how many regions it needs for the next cycle. By the time next cycle ends, there might be a case when there are no empty regions (or the ones that are empty can not take all the live Objects that are supposed to be moved). Where is Eden supposed to move live Objects (Survivor), or, if old region is fragmented - where are live objects supposed to be moved to defragmente? This is what this buffer is for.
It acts as a safety-net (the comments in the code make this far more easier to understand). This is needed in the hopes that it will avoid a Full GC. Because if there are no free regions (or enough) to move live Objects a Full GC needs to happen (most probably followed by a young GC also).
This value is usually known to be small when this message is present in logs. Either increase it, or much better give more heap to the application.
For example in a service adapter you might:
a. have an input data model and an output data model, maybe even immutable, with different classes and use Object Mappers to transform between classes and create some short-lived objects along the way
b. have a single data model, some of the classes might be mutable, but the same object that was created for the input is also sent as output
There are other use-cases when you'd have to choose between clear code with many objects and less clear code with less objects and I would like to know if Garbage Collection still has a weight in this decision.
I should make this a comment as IMO it does not qualify as an answer, but it will not fit.
Even if the answer(s) are going to most probably be - do whatever makes your code more readable (and to be honest I still follow that all the time); we have faced this issue of GC in our code base.
Suppose that you want to create a graph of users (we had to - around 1/2 million) and load all their properties in memory and do some aggregations on them and filtering, etc. (it was not my decision), because these graph objects where pretty heavy - once loaded even with 16GB of heap the JVM would fail with OOM or GC would take huge pauses. And it's understandable - lots of data requires lots of memory, you can't run away from it. The solution proposed and that actually worked was to model that with simple BitSets - where each bit would be a property and a potential linkage to some other data; this is by far not readable and extremely complicated to maintain to this day. Lots of shifts, lots of intrinsics of the data - you have to know at all time what the 3-bit means for example, there's no getter for usernameIncome let's say - you have to do quite a lot shifts and map that to a search table, etc. But it would keep the GC pretty low, at least in the ranges where we were OK with that.
So unless you can prove that GC is taken your app time so much - you probably are even safer simply adding more RAM and increasing it(unless you have a leak). I would still go for clear code like 99.(99) % of the time.
Newer versions of Java have quite sophisticated mechanisms to handle very short-living objects so it's not as bad as it was in the past. With a modern JVM I'd say that you don't need to worry about garbage collection times if you create many objects, which is a good thing since there are now many more of them being created on the fly that this was the case with older versions of Java.
What's still valid is to keep the number of created objects low if the creation is coming with high costs, e.g. accessing a database to retrieve data from, network operations, etc.
As other people have said I think it's better to write your code to solve the problem in an optimum way for that problem rather than thinking about what the garbage collector (GC) will do.
The key to working with the GC is to look at the lifespan of your objects. The heap is (typically) divided into two main regions called generations to signify how long objects have been alive (thus young and old generations). To minimise the impact of GC you want your objects to become eligible for collection while they are still in the young generation (either in the Eden space or a survivor space, but preferably Eden space). Collection of objects in the Eden space is effectively free, as the GC does nothing with them, it just ignores them and resets the allocation pointer(s) when a minor GC is finished.
Rather than explicitly calling the GC via System.gc() it's much better to tune your heap. For example, you can set the size of the young generation using command line options like -XX:NewRatio=n, where n signifies the ratio of new to old (e.g. setting it to 3 will make the ratio of new:old 1:3 so the young generation will be 1 quarter of the heap). Alternatively, you can set the size explicitly using -XX:NewSize=n and -XX:MaxNewSize=m. The GC may resize the heap during collections so setting these values to be the same will keep it at a fixed size.
You can profile your code to establish the rate of object creation and how long your objects typically live for. This will give you the information to (ideally) configure your heap to minimise the number of objects being promoted into the old generation. What you really don't want is objects being promoted and then becoming garbage shortly thereafter.
Alternatively, you may want to look at the Zing JVM from Azul (full disclosure, I work for them). This uses a different GC algorithm, called C4, which enables compaction of the heap concurrently with application threads and so eliminates most of the impact of the GC on application latency.
I've read an extensive amount of documentation about the HotSpot GC of Java SE 6 and 7. When talking about strategies for obtaining contiguous regions of free memory, two 'competing' approaches are presented: that of Evacuation (usually applied on the young gen), where live objects are copied from 'from' to an empty 'to' and that of Compaction (fall-back of CMS), where live object are moved to one side inside a fragmented region to form a contiguous block of used an unused memory.
Both approaches are proportional to the size of the 'live set'. The difference is that evacuation requires x2 times space than the live set, where the compaction does not.
Why do we need the Evacuation technique at all? The amount of copying that needs to be done is the same, however it requires reservation of more heap size, and it does not allow for faster remapping of references.
True: the evacuation can be executed in parallel (where-as compaction cannot, or at least not as easily) but this trait is never mentioned and seems not that important (considering that remapping is much more expensive than moving).
One big problem is that with "evacuation" the vacated space is, indeed vacant, while with "compaction" some other object Y may be moved into the space where object X was. This makes it a lot harder to correct pointers, since one can't simply use the fact that a pointer points to an invalid location to clue the code that it needs to be updated. And one can't store the "forwarding pointer" in the "invalid" location.
This makes GC much less concurrent -- the app must be in "GC freeze" for a longer period of time.
Compaction is more suitable in cases where the number of reclaimable objects is expected to be low(e.g. Tenured generation) because after a few GC cycles the long living objects tend to occupy the lower portion of the heap and hence less work is needed to be done by the collector. If in such a case a copying collector is used that would perform very poorly because almost the same surviving objects from the previous cycles will need to be copied again and again from one location to the other.
Copying is suitable when the number of reclaimable objects is very high(e.g. Young generation) since very few surviving objects needs to be copied. If in such a case compaction is used that may perform poorly because the surviving objects may be scattered across the heap.
Other than that as mentioned in #Hot Licks answer Copying collector allows us to store a forwarding pointer which prevents from running into an infinite loop in case another object from the same "From" space refers to an already moved object.
Also, Compaction can not begin until all the live objects are identified, but live objects can be copied to the new location as soon as they are identified(using multiple threads).
I'm guessing this isn't possible...but here goes. My understanding is that eden space is cheaper to collect than old gen space, especially when you start getting into very large heaps. Large heaps tend to come up with long running applications (server apps) and server apps a lot of the time want to use some kind of caches. Caches with some kind of eviction (LRU) tend to defeat some assumptions that GC makes (temporary objects die quickly). So cache evictions end up filling up old gen faster than you'd like and you end up with a more costly old gen collection.
Now, it seems like this sort of thing could be avoided if java provided a way to mark a reference as about to die (delete keyword)? The difference between this and c++ is that the use is optional. And calling delete does not actually delete the object, but rather is a hint to the GC that it should demote the object back to Eden space (where it will be more easily collected). I'm guessing this feature doesn't exist, but, why not (is there a reason it's a bad idea)?
Actually the eden space is the zone of memory in which objects are newly created. Once an object leaves the eden space it cannot be placed there again, then the GC implementation of Java is so much opaque that there is usually not much to do.
It would break some constrains in any case, the eden space is easily garbage collected in the sense that keep care of removing items that have a short life span. If an object survived enough time then it has to be moved somewhere else, it would be like trying to go against the rules imposed by the GC itself, which is something that is never easily obtainable in Java..
I am building a Java web app, using the Play! Framework. I'm hosting it on playapps.net. I have been puzzling for a while over the provided graphs of memory consumption. Here is a sample:
The graph comes from a period of consistent but nominal activity. I did nothing to trigger the falloff in memory, so I presume this occurred because the garbage collector ran as it has almost reached its allowable memory consumption.
My questions:
Is it fair for me to assume that my application does not have a memory leak, as it appears that all the memory is correctly reclaimed by the garbage collector when it does run?
(from the title) Why is java waiting until the last possible second to run the garbage collector? I am seeing significant performance degradation as the memory consumption grows to the top fourth of the graph.
If my assertions above are correct, then how can I go about fixing this issue? The other posts I have read on SO seem opposed to calls to System.gc(), ranging from neutral ("it's only a request to run GC, so the JVM may just ignore you") to outright opposed ("code that relies on System.gc() is fundamentally broken"). Or am I off base here, and I should be looking for defects in my own code that is causing this behavior and intermittent performance loss?
UPDATE
I have opened a discussion on PlayApps.net pointing to this question and mentioning some of the points here; specifically #Affe's comment regarding the settings for a full GC being set very conservatively, and #G_H's comment about settings for the initial and max heap size.
Here's a link to the discussion, though you unfortunately need a playapps account to view it.
I will report the feedback here when I get it; thanks so much everyone for your answers, I've already learned a great deal from them!
Resolution
Playapps support, which is still great, didn't have many suggestions for me, their only thought being that if I was using the cache extensively this may be keeping objects alive longer than need be, but that isn't the case. I still learned a ton (woo hoo!), and I gave #Ryan Amos the green check as I took his suggestion of calling System.gc() every half day, which for now is working fine.
Any detailed answer is going to depend on which garbage collector you're using, but there are some things that are basically the same across all (modern, sun/oracle) GCs.
Every time you see the usage in the graph go down, that is a garbage collection. The only way heap gets freed is through garbage collection. The thing is there are two types of garbage collections, minor and full. The heap gets divided into two basic "areas." Young and tenured. (There are lots more subgroups in reality.) Anything that is taking up space in Young and is still in use when the minor GC comes along to free up some memory, is going to get 'promoted' into tenured. Once something makes the leap into tenured, it sits around indefinitely until the heap has no free space and a full garbage collection is necessary.
So one interpretation of that graph is that your young generation is fairly small (by default it can be a fairly small % of total heap on some JVMs) and you're keeping objects "alive" for comparatively very long times. (perhaps you're holding references to them in the web session?) So your objects are 'surviving' garbage collections until they get promoted into tenured space, where they stick around indefinitely until the JVM is well and good truly out of memory.
Again, that's just one common situation that fits with the data you have. Would need full details about the JVM configuration and the GC logs to really tell for sure what's going on.
Java won't run the garbage cleaner until it has to, because the garbage cleaner slows things down quite a bit and shouldn't be run that frequently. I think you would be OK to schedule a cleaning more frequently, such as every 3 hours. If an application never consumes full memory, there should be no reason to ever run the garbage cleaner, which is why Java only runs it when the memory is very high.
So basically, don't worry about what others say: do what works best. If you find performance improvements from running the garbage cleaner at 66% memory, do it.
I am noticing that the graph isn't sloping strictly upward until the drop, but has smaller local variations. Although I'm not certain, I don't think memory use would show these small drops if there was no garbage collection going on.
There are minor and major collections in Java. Minor collections occur frequently, whereas major collections are rarer and diminish performance more. Minor collections probably tend to sweep up stuff like short-lived object instances created within methods. A major collection will remove a lot more, which is what probably happened at the end of your graph.
Now, some answers that were posted while I'm typing this give good explanations regarding the differences in garbage collectors, object generations and more. But that still doesn't explain why it would take so absurdly long (nearly 24 hours) before a serious cleaning is done.
Two things of interest that can be set for a JVM at startup are the maximum allowed heap size, and the initial heap size. The maximum is a hard limit, once you reach that, further garbage collection doesn't reduce memory usage and if you need to allocate new space for objects or other data, you'll get an OutOfMemoryError. However, internally there's a soft limit as well: the current heap size. A JVM doesn't immediately gobble up the maximum amount of memory. Instead, it starts at your initial heap size and then increases the heap when it's needed. Think of it a bit as the RAM of your JVM, that can increase dynamically.
If the actual memory use of your application starts to reach the current heap size, a garbage collection will typically be instigated. This might reduce the memory use, so an increase in heap size isn't needed. But it's also possible that the application currently does need all that memory and would exceed the heap size. In that case, it is increased provided that it hasn't already reached the maximum set limit.
Now, what might be your case is that the initial heap size is set to the same value as the maximum. Suppose that would be so, then the JVM will immediately seize all that memory. It will take a very long time before the application has accumulated enough garbage to reach the heap size in memory usage. But at that moment you'll see a large collection. Starting with a small enough heap and allowing it to grow keeps the memory use limited to what's needed.
This is assuming that your graph shows heap use and not allocated heap size. If that's not the case and you are actually seeing the heap itself grow like this, something else is going on. I'll admit I'm not savvy enough regarding the internals of garbage collection and its scheduling to be absolutely certain of what's happening here, most of this is from observation of leaking applications in profilers. So if I've provided faulty info, I'll take this answer down.
As you might have noticed, this does not affect you. The garbage collection only kicks in if the JVM feels there is a need for it to run and this happens for the sake of optimization, there's no use of doing many small collections if you can make a single full collection and do a full cleanup.
The current JVM contains some really interesting algorithms and the garbage collection itself id divided into 3 different regions, you can find a lot more about this here, here's a sample:
Three types of collection algorithms
The HotSpot JVM provides three GC algorithms, each tuned for a specific type of collection within a specific generation. The copy (also known as scavenge) collection quickly cleans up short-lived objects in the new generation heap. The mark-compact algorithm employs a slower, more robust technique to collect longer-lived objects in the old generation heap. The incremental algorithm attempts to improve old generation collection by performing robust GC while minimizing pauses.
Copy/scavenge collection
Using the copy algorithm, the JVM reclaims most objects in the new generation object space (also known as eden) simply by making small scavenges -- a Java term for collecting and removing refuse. Longer-lived objects are ultimately copied, or tenured, into the old object space.
Mark-compact collection
As more objects become tenured, the old object space begins to reach maximum occupancy. The mark-compact algorithm, used to collect objects in the old object space, has different requirements than the copy collection algorithm used in the new object space.
The mark-compact algorithm first scans all objects, marking all reachable objects. It then compacts all remaining gaps of dead objects. The mark-compact algorithm occupies more time than the copy collection algorithm; however, it requires less memory and eliminates memory fragmentation.
Incremental (train) collection
The new generation copy/scavenge and the old generation mark-compact algorithms can't eliminate all JVM pauses. Such pauses are proportional to the number of live objects. To address the need for pauseless GC, the HotSpot JVM also offers incremental, or train, collection.
Incremental collection breaks up old object collection pauses into many tiny pauses even with large object areas. Instead of just a new and an old generation, this algorithm has a middle generation comprising many small spaces. There is some overhead associated with incremental collection; you might see as much as a 10-percent speed degradation.
The -Xincgc and -Xnoincgc parameters control how you use incremental collection. The next release of HotSpot JVM, version 1.4, will attempt continuous, pauseless GC that will probably be a variation of the incremental algorithm. I won't discuss incremental collection since it will soon change.
This generational garbage collector is one of the most efficient solutions we have for the problem nowadays.
I had an app that produced a graph like that and acted as you describe. I was using the CMS collector (-XX:+UseConcMarkSweepGC). Here is what was going on in my case.
I did not have enough memory configured for the application, so over time I was running into fragmentation problems in the heap. This caused GCs with greater and greater frequency, but it did not actually throw an OOME or fail out of CMS to the serial collector (which it is supposed to do in that case) because the stats it keeps only count application paused time (GC blocks the world), application concurrent time (GC runs with application threads) is ignored for those calculations. I tuned some parameters, mainly gave it a whole crap load more heap (with a very large new space), set -XX:CMSFullGCsBeforeCompaction=1, and the problem stopped occurring.
Probably you do have memory leaks that's cleared every 24 hours.