Develop Reference

Unexpected Full GC (Ergonomic)

I'm running application that consists of 3 stages: load initial data, update it using POST http requests, serve it using GET requests.
After all data is set up (initial load + POST updates) I'm calling System.gc() and I see
[Full GC (System.gc()) 2051102K->1082623K(3298304K), 13.3521960 secs]
So I expect that I need about 1000M for old gen and run my app with such setup (at 4G RAM docker container):
java -XX:MaxNewSize=2550m -XX:NewSize=2550m -Xms3750m -Xmx3750m -XX:+PrintGCTimeStamps -XX:+PrintGC -cp my.jar my.My
So I expect that from 3750M of heap 2550M will be used for new space to serve GET requests and 1200M will be for old gen.
During the work of my application I have
[Full GC (Ergonomics) 2632447K->1086729K(3298304K), 2.3360398 secs]
So it looks very strange for me that full GC is triggered. My understanding is that because GET requests don't add any long-living data it should be just Minor GC when new generation is full. So I don't understand
1) Why it is running Full GC (Ergonomics) - does it want to change generation layout?
2) Why it is running when only 2632447K of 3298304K are occupied?

A few points:
GET/POST has nothing to do with it, the logic that you implemented for GET/POST is what matters... e.g. the GET method might have a memory leak...
Growing memory doesn't necessarily reduce GC (it doesn’t work in linearly)
GC usually freezes for ~1 sec per live GB, so the fact that you're having 13 secs freeze for 1 GB worths checking
The next step should be enabling GC log and analyzing it to have a better understanding of the issue

JVM heap used percentage - when to generate alert

We have an application which is deployed on Tomcat 8 application server and currently monitoring server (Zabbix) is configured to generate alert if the heap memory is 90% utilized.
There were certain alerts generated which prompted us to do heap dump analysis. Nothing really came out of heap dump, there was no memory leak. There were lot of unreachable object which were not cleaned up because of no GC.
JVM configurations:
-Xms8192m -Xmx8192m -XX:PermSize=128M -XX:MaxPermSize=256m
-XX:+UseParallelGC -XX:NewRatio=3 -XX:+PrintGCDetails
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/app/apache-tomcat-8.0.33
-XX:ParallelGCThreads=2
-Xloggc:/app/apache-tomcat-8.0.33/logs/gc.log
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps -XX:GCLogFileSize=50m -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=30
We tried running garbage collection manually using jcmd command and it cleared up the memory. GC logs after running jcmd:
2016-11-04T03:06:31.751-0400: 1974627.198: [Full GC (System.gc()) [PSYoungGen: 18528K->0K(2049024K)] [ParOldGen: 5750601K->25745K(6291456K)] 5769129K->25745K(8340480K), [Metaspace: 21786K->21592K(1069056K)], 0.1337369 secs] [Times: user=0.19 sys=0.00, real=0.14 secs]
Questions:
Is there any configuration above due to which GC is not running automatically.
What is the reason of this behavior? I understand that Java will do GC when it needs to. But, if it is not running GC even when heap is 90% utilized, what should be the alert threshold (and if it even makes sense to have any alert based on heap utilization).

When the garbage collector decides to collect differs per garbage collector. I have not been able to find any hard promises on when your (Parallel GC) garbage collector runs. Many Garbage collectors also tune on several different variables, which can influence when it will run.
As you have noted yourself, your application can have high heap usage and still run fine. What you are looking for in an application is that the Garbage Collector is still efficient. Meaning it can clean up quiet a lot of garbage in a single run.
Some aspects of garbage collection
Most garbage collectors have two or more strategies, one for 'young' objects and one for 'old' objects. When a young object has not been collected in the latest (several) collects, it becomes an old object. The idea behind this is that if an object has not been collected it probably wont be collected next time either. (Most objects either live really short, or really long). The garbage collector does a very efficient, but not perfect cleaning of the young objects. When that doesn't free up enough data, then a more costly garbage collection is done on all (young en old) objects.
This will often generate a saw tooth (taken from this site):
Here you see many small drops in heap size and a slowly growing heap. Every now and then a large collection is done, and there is a large drop. The actually 'used' memory is the amount of memory left after a large collection.
Aspects to measure
This leads to the following aspects you can look at when determining the health of your application:
The amount of time spent by you application garbage collecting (both in total and as a percentage of CPU time).
The amount of memory available right after a garbage collect.
The a quick increment in the number of large garbage collects.
In most cases you will need monitor the behavior of your application under load, to see what are good values for you.
The parallel garbage collector uses a similar condition to determine if all is still well:
If more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered, then an OutOfMemoryError is thrown.
All of these statistic you can see nicely using VisualVM and Jconsole. I am not sure which you can use as triggers in your monitoring tools

Xmx and gc log data not matching

I ran a test program (adds list with string in a infinite loop) using below jvm arguments
java -XX:+PrintGCDetails -Xloggc:gc.log Test -Xmx=1k -Xms=1k
and got the below exception
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
In the gc log i see the below collection as the last entry
11.242: [Full GC (Allocation Failure) [PSYoungGen: 0K->0K(92160K)] [ParOldGen: 274084K->274072K(371712K)] 274084K->274072K(**463872K**), [Metaspace: 2522K->2522K(1056768K)], 3.0296130 secs] [Times: user=3.28 sys=0.00, real=3.02 secs]
If min and max is 1k how come the heap memory available is showing as 463872K?
Plus in the oracle site
https://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/jrdocs/refman/optionX.html
I see a note ,
Note: -Xmx does not limit the total amount of memory that the JVM can use.
What does this means?

You have specified Xmx and Xms parameters after the name of your class. Thus java executable command interprets them as parameters to your class, not as options to configure JVM. The correct way would be:
java -Xmx1k -Xms1k -XX:+PrintGCDetails -Xloggc:gc.log Test
Also note, that the correct format is -Xmx1k, not -Xmx=1k.
But please not, that JVM will not start with such low value of Xms.

#Nikem responded correctly and I would not like to respond to that part of the question. However, I would like to add something to your second question -
Note: -Xmx does not limit the total amount of memory that the JVM can use. What does this imply?
-Xmx only limits the memory available for consumption by your application heap.
But JVM will need memory space for PermGen and stack sizes which are not accounted for with the -Xmx parameter.

Optimizing Tomcat / Garbage Collection

Our server has 128GB of RAM and 64 cores, running Tomcat 7.0.30 and Oracle jdk1.6.0_38, on CentOS 6.3.
Every 60 minutes we're seeing garbage collection that was taking 45 - 60 seconds. Adding -XX:-UseConcMarkSweepGC increased page load times by about 10% but got that down to about 3 seconds, which is an acceptable trade-off.
Our config:
-Xms30g
-Xmx30g
-XX:PermSize=8g
-XX:MaxPermSize=8g
-Xss256k
-XX:-UseConcMarkSweepGC
We set the heap at 30 GB to keep 32 bit addressing (I read that above 32 GB the 64 bit addressing takes up more memory, so you have to go to about 48 GB to see improvements).
Using VisualGC I can see that the Eden space is cycling through every 30 - 60 minutes, but not much happens with the Survivor 0, Survivor 1, Old Gen, and Perm Gen.
We have a powerful server. What other optimizations can we make to further decrease the 3 second GC time?
Any recommendations to improve performance or scaling?
Any other output or config info that would help?

It might sound counter-intuitive, but have you tried allocating a lot less memory? E.g. do you really need a 30G heap? In case you can get along with 4G or even less: Garbage collection might be more frequent, but when it happens it will be a lot faster. Typically I find this more desirable than allocating a lot of memory, suffering from the time it takes to clean it up.
Even if this will not help you because you really need 30G of memory, others might come along with a similar problem and they might benefit from allocating less.

Seems that you need Incremental GC to reduce pauses:
-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
and for tracing without visualgc this always went well for me (output in catalina.out):
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
2013-01-05T22:52:13.954+0100: 15918369.557: [GC 15918369.557: [DefNew:
65793K->227K(98304K), 0.0031220 secs] 235615K->170050K(491520K),
0.0033220 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
After you can play with this:
-XX:NewSize=ABC -XX:MaxNewSize=ABC
-XX:SurvivorRatio=ABC
-XX:NewRatio=ABC
Reference: Virtual Machine Garbage Collection Tuning

Shrinking survivor spaces lead to continuous full GC

I've had this troubling experience with a Tomcat server, which runs:
our Hudson server;
a staging version of our web application, redeployed 5-8 times per day.
The problem is that we end up with continuous garbage collection, but the old generation is nowhere near to being filled. I've noticed that the survivor spaces are next to inexisting, and the garbage collector output is similar to:
[GC 103688K->103688K(3140544K), 0.0226020 secs]
[Full GC 103688K->103677K(3140544K), 1.7742510 secs]
[GC 103677K->103677K(3140544K), 0.0228900 secs]
[Full GC 103677K->103677K(3140544K), 1.7771920 secs]
[GC 103677K->103677K(3143040K), 0.0216210 secs]
[Full GC 103677K->103677K(3143040K), 1.7717220 secs]
[GC 103679K->103677K(3143040K), 0.0219180 secs]
[Full GC 103677K->103677K(3143040K), 1.7685010 secs]
[GC 103677K->103677K(3145408K), 0.0189870 secs]
[Full GC 103677K->103676K(3145408K), 1.7735280 secs]
The heap information before restarting Tomcat is:
Attaching to process ID 10171, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 14.1-b02
using thread-local object allocation.
Parallel GC with 8 thread(s)
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 3221225472 (3072.0MB)
NewSize = 2686976 (2.5625MB)
MaxNewSize = 17592186044415 MB
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 268435456 (256.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 1073479680 (1023.75MB)
used = 0 (0.0MB)
free = 1073479680 (1023.75MB)
0.0% used
From Space:
capacity = 131072 (0.125MB)
used = 0 (0.0MB)
free = 131072 (0.125MB)
0.0% used
To Space:
capacity = 131072 (0.125MB)
used = 0 (0.0MB)
free = 131072 (0.125MB)
0.0% used
PS Old Generation
capacity = 2147483648 (2048.0MB)
used = 106164824 (101.24666595458984MB)
free = 2041318824 (1946.7533340454102MB)
4.943684861063957% used
PS Perm Generation
capacity = 268435456 (256.0MB)
used = 268435272 (255.99982452392578MB)
free = 184 (1.7547607421875E-4MB)
99.99993145465851% used
The relevant JVM flags passed to Tomcat are:
-verbose:gc -Dsun.rmi.dgc.client.gcInterval=0x7FFFFFFFFFFFFFFE -Xmx3g -XX:MaxPermSize=256m
Please note that the survivor spaces are sized at about 40 MB at startup.
How can I avoid this problem?
Updates:
The JVM version is
$ java -version
java version "1.6.0_15"
Java(TM) SE Runtime Environment (build 1.6.0_15-b03)
Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02, mixed mode)
I'm going to look into bumping up the PermGen size and seeing if that helps - probably the sizing of the survivor spaces was unrelated.

The key is probably PS Perm Generation which is at 99.999% (only 184 bytes out of 256***MB*** free).
Usually, I'd suggest that you give it more perm gen but you already gave it 256MB which should be plenty. My guess is that you have a memory leak in some code generation library. Perm Gen is mostly used for bytecode for classes.

It's very easy to have ClassLoader leaks - all it takes is a single object loaded through the ClassLoader being referred by an object not loaded by it. A constantly redeployed app will then quickly fill PermGenSpace.
This article explains what to look out for, and a followup describes how to diagnose and fix the problem.

I think this is not that uncommon for an application server that gets continuously deployed to. The perm gen space, which is full for you, is where classes go. Keep in mind that JSPs are compiled as Java classes, and when you change a JSP, a new class gets generated and loaded.
We have had this problem, and our solution is to have the app server restart occasionally.
This is what I'd do:
Deploy Hudson to a separate server from your staging server
Configure Hudson to restart your staging server from time to time. You can either do this one of two ways:
Restart periodically (e.g., every night at midnight, regardless of if there's build activity); or
Have the web app deployment job trigger the server restart job. If you do this make sure there's a really long quiet period for the restart job (we set ours to 2 hours), so that you don't get a server restart for every build (i.e., if two web app deployments happen within 2 hours, they'll only trigger one server restart).

The flag -XX:SurvivorRatio sets the ratio between Eden and the survivor spaces. According to the JDK 1.5 tuning doc, the default value is 32, which gives a 1:32 ratio. This is in accordance with what you're seeing. It seems incredibly small to me, although I understand that only a very small number of objects are expected to make their way from Eden to the survivor space.
So, assuming that you have a lot of long-lived objects, you should decrease the survivor ratio. The risk is that you only have those long-lived objects during a startup phase, and so are limiting the Eden size. For a testing server, I doubt this is going to be an issue.
I'd probably also reduce the size of the Eden space, by increasing -XX:NewRatio (the default is 3). My gut says that a hundred MB or so is sufficient for the young generation, and you'll just be increasing the cost of garbage collection to have such a large amount of space allocated (ie, object will live in Eden far too long). But that's just instinct, and should definitely be validated for your environment.
And a semi-related comment, after reading other replies: if you're not seeing errors for running out of permgen space, don't spend your time fiddling with it. The permgen is managed separately from the rest of the heap.