Develop Reference

Empty Heap Dump after OutOfMemoryException

I'm getting an OutOfMemoryException on Tomcat7 and openjdk 1.8.0_162.
A heap dump is being created but when analyzing with Memory Analyzer (MAT) no problems are found and the tool reports lots of free heap space.
I'm aware what GC overhead limit exceeded means I just cannot figure out what is being done at that time. The heap dump would normally show me what objects taking up so much space.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to /var/log/tomcat7/java_pid13747.hprof ...
Heap dump file created [4694238678 bytes in 50.790 secs]
#
# java.lang.OutOfMemoryError: GC overhead limit exceeded
some log lines from the gc.log:
...
2018-04-17T10:12:29.208+0200: 6436.957: [Full GC (Ergonomics) [PSYoungGen: 465920K->464244K(931840K)] [ParOldGen: 2796166K->2796166K(2796544K)] 3262086K->3260410K(3728384K), [Metaspace: 233392K->233392K(1306624K)], 6.6466544 secs] [Times: user=25.17 sys=0.03, real=6.65 secs]
2018-04-17T10:12:35.866+0200: 6443.614: [Full GC (Ergonomics) [PSYoungGen: 465920K->464395K(931840K)] [ParOldGen: 2796166K->2796165K(2796544K)] 3262086K->3260560K(3728384K), [Metaspace: 233396K->233396K(1306624K)], 6.6379297 secs] [Times: user=25.01 sys=0.06, real=6.64 secs]
2018-04-17T10:12:42.516+0200: 6450.265: [Full GC (Ergonomics) [PSYoungGen: 465920K->464702K(931840K)] [ParOldGen: 2796165K->2796165K(2796544K)] 3262085K->3260867K(3728384K), [Metaspace: 233403K->233400K(1306624K)], 6.9424274 secs] [Times: user=25.14 sys=0.08, real=6.94 secs]
...
MAT chart (grey is free space...):
How to approach this?
I suspect that some resource heavy operation is being done but the heap dump is empty...

Garbage collector not able to clear garbage, causes full frequent gc

We use parallel gc with 1.7.0_71.
java -XX:+PrintCommandLineFlags -XX:+PrintGCDetails -version
-XX:InitialHeapSize=258222272 -XX:MaxHeapSize=4131556352 -XX:+PrintCommandLineFlags -XX:+PrintGCDetails -XX:+UseCompressedOops -XX:+UseParallelGC
java version "1.7.0_71"
Java(TM) SE Runtime Environment (build 1.7.0_71-b14)
Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)
We are seeing huge garbage created in our application every few minutes under decent load . Minor gc triggers every few seconds and major gc triggers every 2 minutes. On analysis we found that 1.8 million char array objects each of size 173K is left in heap. Minor gc is not able to recover them . On taking heapdump ,we found a lot of objects in remainder section in Eclipe MAT. MAT histogram shows lots of char arrays (they are rendered html's),however all incoming references and merge paths to gc root for the char array is unreachable, but only full GC is only able to recover them not minor GC. Why minor GC is not able to recover them when there are no referenced objects ? Eclipse MAT pictures attached.
Please see all references are unreachable.
There were some reference to request objects, cleared them by setting nulls.
Before the fix there were hanging references to HttpServletRequest objects
All images related to the issue:
https://www.dropbox.com/sh/qgsitzb7x27j8kc/AABoQwR1qPwTPiDtO6B0_Pm7a?dl=0
Overall heap
Heap Histogram view
Now the question is
Do you think this is issue with garbage collector? We upgraded JBoss 7.1.1 to wildfly 8.2.0.Final. No changes in gc strategy and JDK version. Why gc is not able to reclaim memory where it is pointing to unreachable references.
We can decrease XX:NewRatio to 1. However we are no sure this will work as full gc happens frequently
DO you think moving to G1 GC will help ? Will there be decrease in throughput ? What are best options available for G1 GC. Our heap size -Xms : 1024m , -Xmx 2048m. Perm gen 512m
We are not seeing memory leaks, no out of memory error.Attaching full gc log outputs
2015-05-10 19:32:41 IST| 459.939: [Full GC [PSYoungGen: 8123K->0K(680960K)] [ParOldGen: 782136K->359065K(766464K)] 790260K->359065K(1447424K) [PSPermGen: 202932K->202930K(441344K)], 1.0738240 secs] [Times: user=3.37 sys=0.01, real=1.07 secs]
2015-05-10 19:32:42 IST| 462.306: [GC [PSYoungGen: 672768K->10534K(685056K)] 1031833K->369600K(1451520K), 0.0450800 secs] [Times: user=0.15 sys=0.00, real=0.04 secs]
2015-05-10 19:32:44 IST| 463.641: [GC [PSYoungGen: 682790K->9093K(685568K)] 1041856K->373085K(1452032K), 0.0570820 secs] [Times: user=0.16 sys=0.00, real=0.06 secs]
2015-05-10 19:32:45 IST| 464.936: [GC [PSYoungGen: 681349K->9812K(686080K)] 1045341K->377511K(1452544K), 0.0439060 secs] [Times: user=0.12 sys=0.00, real=0.04 secs]
2015-05-10 19:32:46 IST| 466.283: [GC [PSYoungGen: 683092K->10733K(686080K)] 1050791K->383554K(1452544K), 0.0464700 secs] [Times: user=0.14 sys=0.00, real=0.05 secs]
2015-05-10 19:32:48 IST| 467.659: [GC [PSYoungGen: 684013K->11283K(685568K)] 1056834K->388651K(1452032K), 0.1381130 secs] [Times: user=0.30 sys=0.00, real=0.14 secs]
2015-05-10 19:32:50 IST| 469.734: [GC [PSYoungGen: 684051K->9652K(686080K)] 1061419K->393759K(1452544K), 0.0466800 secs] [Times: user=0.13 sys=0.00, real=0.05 secs]
2015-05-10 19:32:51 IST| 471.087: [GC [PSYoungGen: 682420K->11253K(685568K)] 1066527K->400087K(1452032K), 0.0589180 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
2015-05-10 19:32:52 IST| 472.325: [GC [PSYoungGen: 684021K->7957K(686080K)] 1072855K->403018K(1452544K), 0.0436140 secs] [Times: user=0.13 sys=0.00, real=0.04 secs]
2015-05-10 19:32:54 IST| 473.606: [GC [PSYoungGen: 680725K->9177K(685056K)] 1075786K->406493K(1451520K), 0.0524990 secs] [Times: user=0.13 sys=0.00, real=0.05 secs]
2015-05-10 19:34:34 IST| 573.526: [GC [PSYoungGen: 684217K->10956K(686080K)] 1440629K->771626K(1452544K), 0.0416620 secs] [Times: user=0.14 sys=0.00, real=0.04 secs]
2015-05-10 19:34:34 IST| 573.568: [Full GC [PSYoungGen: 10956K->0K(686080K)] [ParOldGen: 760670K->364958K(818688K)] 771626K->364958K(1504768K) [PSPermGen: 203069K->203069K(420864K)], 0.8001740 secs] >[Times: user=2.46 sys=0.01, real=0.80 secs]
2015-05-10 19:34:36 IST| 575.600: [GC [PSYoungGen: 674304K->10465K(686592K)] 1039262K->375423K(1505280K), 0.0410330 secs] [Times: user=0.13 sys=0.00, real=0.04 secs]
2015-05-10 19:36:35 IST| 694.277: [GC [PSYoungGen: 684413K->9342K(687104K)] 1490469K->820033K(1505792K), 0.2160320 secs] [Times: user=0.55 sys=0.08, real=0.21 secs]
2015-05-10 19:36:36 IST| 695.664: [GC [PSYoungGen: 684670K->8323K(687104K)] 1495361K->823380K(1505792K), 0.0454050 secs] [Times: user=0.11 sys=0.00, real=0.05 secs]
2015-05-10 19:36:37 IST| 695.710: [Full GC [PSYoungGen: 8323K->0K(687104K)] [ParOldGen: 815056K->363295K(838144K)] 823380K->363295K(1525248K) [PSPermGen: 203095K->203095K(401920K)], 0.8133080 secs] [Times: user=2.43 sys=0.01, real=0.81 secs]
2015-05-10 19:36:38 IST| 697.669: [GC [PSYoungGen: 675328K->10586K(686592K)] 1038623K->373882K(1524736K), 0.0436000 secs] [Times: user=0.13 sys=0.00, real=0.04 secs]
...
....

Why minor GC is not able to recover them when there are no referenced objects ?
Most likely because they are in the old generation. Minor GCs only dispose of unreachable objects in the young generation.
This usually happens when the object lives longer than it can remain in the young generation due to tenuring. E.g. if caches are involved that hold onto results for a while or when the request lifetime exceeds that of the GC interval * tenuring threshold.
-XX:+PrintTenuringDistribution may prove informative.
First you could try simply providing a pause time goal via -XX:MaxGCPauseMillis=.... ParallelGC might be able to meet it.
If that doesn't help you could refactor your code to reduce object lifetimes or reduce allocation rates to make minor GCs less frequent.
Note that foremost ParallelGC is a throughput collector, in terms of CPU cycles it's more efficient than concurrent collectors but it usually won't be able to meet as low pause time goals as those.
DO you think moving to G1 GC will help ?
If you are concerned about pause times, most likely. You might also want to try CMS.
If you want to try out G1 you should probably switch to java 8, its heuristics have improved a lot over time and it is still maturing.
Will there be decrease in throughput ?
possibly. it depends on whether there's spare CPU capacity and how you define/measure throughput.
And even under less favorable circumstances the decrease might not be significant.
What are best options available for G1 GC.
G1 is supposed to self-tuning (beyond the user-provided pause and throughput goals that also apply to ParallelGC). So just enable it and see if it provides acceptable performance.

Similar issue happened with us in >NET. A reference leaked into Gen1 - and that aged into Gen 2, finally causing the accumulation of RAM into 30Gb+ size.
Although not related to JAVA directly, we solved the similar problem by building a custom memory manager which sub-allots byte[]. This allowed us to store hundreds of millions of objects in-process for days without loading GC at all. The solution is great for caching and far faster than out-of-process Redis/memcache. It serializes objects with super-fast serializer, that way "references" are turned into "Pointer" (our struct of two ints), this way the GC is not overloaded with scanning 1000000000 objects
See this:
https://www.youtube.com/watch?v=Dz_7hukyejQ

Java process taking more memory than its max heap size

I am running two tomcat servers in a single machine(Ram size 16GB), with the memory setting given below.
JAVA_MEM_OPTIONS="-Xmx4096M -Xms4096M -XX:NewSize=512M -XX:MaxNewSize=512M -XX:PermSize=256M -XX:MaxPermSize=256M -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:SoftRefLRUPolicyMSPerMB=5 -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC"
Top result shows 46.5%(7.44 GB) & 39.2%(6.24 GB) of memory are used by java process.
Tasks: 120 total, 1 running, 119 sleeping, 0 stopped, 0 zombie
Cpu(s): 7.8%us, 0.4%sy, 0.0%ni, 90.9%id, 0.2%wa, 0.0%hi, 0.7%si, 0.0%st
Mem: 16424048k total, 16326072k used, 97976k free, 28868k buffers
Swap: 1959920k total, 1957932k used, 1988k free, 1082276k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26275 user1 25 0 8007m 7.3g 7832 S 13.6 46.5 785:31.89 java
28050 user2 25 0 7731m 6.1g 13m S 9.0 39.2 817:10.47 java
GC log shows each java process is using only 4GB of memory. Once the system memory usage goes up its never coming down.
user1 - java process
89224.492: [GC 89224.492: [ParNew: 443819K->17796K(471872K), 0.0367070 secs] 2032217K->1608465K(4141888K), 0.0368920 secs] [Times: user=0.13 sys=0.00, real=0.03 secs]
89228.247: [GC 89228.247: [ParNew: 437252K->22219K(471872K), 0.0607080 secs] 2027921K->1615327K(4141888K), 0.0609240 secs] [Times: user=0.15 sys=0.00, real=0.06 secs]
user2 - java process
89202.170: [GC 89202.170: [ParNew: 444989K->22909K(471872K), 0.0510290 secs] 2361057K->1945258K(4141888K), 0.0512370 secs] [Times: user=0.19 sys=0.00, real=0.05 secs]
89207.894: [GC 89207.894: [ParNew: 442365K->15912K(471872K), 0.0422190 secs] 2364714K->1945162K(4141888K), 0.0424260 secs] [Times: user=0.15 sys=0.00, real=0.04 secs]
How java process can use this much memory when heap memory is set to 4GB. How to debug the cause for the problem.
PS : At times i am executing shell script from java code. Will it leads to this sort of problem ?

There are several cases:
Having a large number of threads. Thread stack space is allocated outside of heap/perm space. To set a memory allocated per thread use -Xss option.
Using JNI. Take a look at this question Java app calls C++ DLL via JNI; how best to allocate memory?
Probably there are more, in practice, first case most likely is the reason.

Why would a java heap allocation resizing cause an OOME?

Why would a java heap allocation resizing cause an OOME?
We are seeing OutOfMemoryExceptions in our logs, and they seem to coincide with the java heap commit size growing from ~1G to ~2.4G. Despite the error message, it does not appear that we are running out of heap space. Aside from the exception being thrown (and heap dump generated), the resize seems to eventually succeed and the application continues on without a problem (with a ~2.4G heap commit size).
Here's an example of the log output:
INFO | jvm 1 | 2013/08/16 12:08:05 | [GC [PSYoungGen: 328000K->2997K(339200K)] 645686K->320683K(1038272K), 0.0101580 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
INFO | jvm 1 | 2013/08/16 12:09:14 | [GC [PSYoungGen: 331509K->3487K(338816K)] 649195K->322153K(1037888K), 0.0115600 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
INFO | jvm 1 | 2013/08/16 12:09:59 | [GC [PSYoungGen: 331999K->2928K(340032K)] 650665K->322608K(1039104K), 0.0099300 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
INFO | jvm 1 | 2013/08/16 12:10:48 | [GC [PSYoungGen: 333104K->2723K(339648K)] 652784K->323240K(1038720K), 0.0100130 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
INFO | jvm 1 | 2013/08/16 12:11:28 | [GC [PSYoungGen: 332885K->3884K(340864K)] 653402K->325089K(1039936K), 0.0106250 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
INFO | jvm 1 | 2013/08/16 12:11:39 | [GC [PSYoungGen: 23694K->463K(340352K)] 344899K->323656K(2437504K), 0.0070330 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
INFO | jvm 1 | 2013/08/16 12:11:39 | [GC [PSYoungGen: 463K->0K(340608K)] 323656K->323592K(2437760K), 0.0044440 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
INFO | jvm 1 | 2013/08/16 12:11:39 | [Full GC
INFO | jvm 1 | 2013/08/16 12:11:40 | [PSYoungGen: 0K->0K(340608K)] [PSOldGen: 323592K->323592K(699072K)] 323592K->323592K(1039680K) [PSPermGen: 159297K->159297K(262144K)], 1.2076900 secs] [Times: user=1.20 sys=0.00, real=1.21 secs]
INFO | jvm 1 | 2013/08/16 12:11:40 | [GC [PSYoungGen: 0K->0K(340736K)] 323592K->323592K(2437888K), 0.0046330 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
INFO | jvm 1 | 2013/08/16 12:11:40 | [Full GC
INFO | jvm 1 | 2013/08/16 12:11:42 | [PSYoungGen: 0K->0K(340736K)] [PSOldGen: 323592K->279953K(744512K)] 323592K->279953K(1085248K) [PSPermGen: 159297K->159062K(262144K)], 1.7593100 secs] [Times: user=1.75 sys=0.00, real=1.76 secs]
INFO | jvm 1 | 2013/08/16 12:11:42 | java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2013/08/16 12:11:42 | Dumping heap to java_pid28908.hprof ...
INFO | jvm 1 | 2013/08/16 12:11:48 | Heap dump file created [463314899 bytes in 6.037 secs]
INFO | jvm 1 | 2013/08/16 12:12:36 | [GC [PSYoungGen: 331840K->6044K(352192K)] 611793K->285998K(2449344K), 0.0164060 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
INFO | jvm 1 | 2013/08/16 12:13:28 | [GC [PSYoungGen: 352156K->6161K(364160K)] 632110K->286114K(2461312K), 0.0152330 secs] [Times: user=0.02 sys=0.01, real=0.01 secs]
INFO | jvm 1 | 2013/08/16 12:14:47 | [GC [PSYoungGen: 364113K->6575K(374144K)] 644066K->288169K(2471296K), 0.0179930 secs] [Times: user=0.02 sys=0.01, real=0.02 secs]
Notice that immediately prior to the OOME, the total committed heap oscillates between 1GB and 2.4GB. We can see that it was pretty steady at 1GB beforehand and pretty steady at 2.4GB afterwards.
Among the javaopts for this 1.6.0._24 JVM are:
-Xmx3072m
-XX:+HeapDumpOnOutOfMemoryError
-XX:-UseGCOverheadLimit
-verbose:gc
-Xss256k
-XX:MaxPermSize=256m
-server
-XX:+PrintGCDetails
The JVM is running 1.6.0._24. We can't change versions now, but will have a window to do so in the next month or two. If 1.6.0_45 is that much more stable, we will aim to switch to that. We're currently testing it.
The machine only has 4GB of total system ram. In addition, there is a small RAM-disk in use as well. I'm concerned that the Xmx setting is already too high for this environment.
This is puzzling us, as it doesn't seem like the heap usage is very big at the time of the exception. Why are we getting this OOME?
UPDATE: We are attempting to prevent this condition by setting the initial memory (Xms) equal to the maximum memory (Xmx). So far, these experiments have been promising though we've yet to introduce the change in production. It still doesn't explain why the OOME occurred in the first place, though it does seem to indicate that the OOME can be avoided without increasing the max heap size (or reducing the app memory footprint). So the mystery remains of why the heap resizing caused an OOME?

For reading you logs, it appear you have a very large burst of activity, most like objects large enough to go straight into tenured/old gen. I still suggest you increase you max memory to see how your application behaves as the OOME is likely to give you confusing stats.
This suggests heavy early promotion. The "GC" is a minor collection which it appears that every object is needed, triggering a Full GC which finds some tenured objects which can be dropped. The GC works best when the young object die in the Eden space, but it appears most of your objects are dying in tenured space.
One way to test this is to make the max heap space much larger. If you can try a 24 GB heap or 80% of you main memory see how it behaves then. e.g. try -Xmx24g if you have 32 GB of memory. From these numbers, it appears you want an Eden size of at least 5 GB.
If this is not an option, I suggest you use a memory profiler to reduce the memory consumption by a factor of at least 3x.
I would check you have a recent version of Java 6 like update 45. There were significant performance improvement between update 18 and 26.

Java ConcurrentMarkSweep garbage collector not happening

Environment Details:
OS: Linux RedHat
Java: JRE 6 Update 21
I am using following GC setting for my app.
-server -d64 -Xms8192m -Xmx8192m -javaagent:lib/instrum.jar -XX\:MaxPermSize=256m -XX\:+UseParNewGC -X\:+ParallelRefProcEnabled -XX\:+UseConcMarkSweepGC -XX\:MaxGCPauseMillis=250 -XX\:+CMSIncrementalMode -XX\:+CMSIncrementalPacing -XX\:+CMSParallelRemarkEnabled -verbose\:gc -Xloggc\:/tmp/my-gc.log -XX\:DisableExplicitGC -XX\:+PrintGCTimeStamps -XX\:+PrintGCDetails -XX\:+UseCompressedOops
With there setting, there is single Full GC at the begining of application
2.946: [Full GC 2.946: [CMS: 0K->7394K(8111744K), 0.1364080 secs] 38550K->7394K(8360960K), [CMS Perm : 21247K->21216K(21248K)], 0.1365530 secs] [Times: user=0.10 sys=0.04, real=0.14 secs]
Which is followed by a 4-5 successful of CMS collections, But after this there is no trace of CMS in logs, there are entries on only minor collections.
379022.293: [GC 379022.293: [ParNew: 228000K->4959K(249216K), 0.0152000 secs] 7067945K->6845720K(8360960K) icms_dc=0 , 0.0153940 secs]
The heap is growing continuously and it has reached 7GB. We have to restart the application as we can not afford OOM or any kind of breakdown in production system.
I am not able to understand as to why CMS collector has stopped cleaning. Any clues/suggestions are welcome. Thanks in Advance.
======================================================================================
Updated 23rd Jan.
Thanks everyone for the responses till now. I have setup the application in test environment and tested the app with following set of JVM options:
Option #1
-server -d64 -Xms8192m -Xmx8192m -javaagent\:instrum.jar -XX\:MaxPermSize\=256m -XX\:+UseParNewGC -XX\:+UseConcMarkSweepGC -verbose\:gc -Xloggc\:my-gc.log -XX\:+PrintGCTimeStamps -XX\:+PrintGCDetails
Option #2
-server -d64 -Xms8192m -Xmx8192m -javaagent\:instrum.jar -XX\:MaxPermSize\=256m -XX\:+UseParNewGC -XX\:+UseConcMarkSweepGC -verbose\:gc -Xloggc\:my-gc.log -XX\:+DisableExplicitGC -XX\:+PrintGCTimeStamps -XX\:+PrintGCDetails
I ran the test with both settings for 2 days in parallel. These are my observations:
Option #1
The heap memory is stable but there are 90 ConcurrentMarkSweep collections and JVM spent 24 minutes. That’s too high. And I see following lines in GC logs and the pattern continues every one hour...
318995.941: [GC 318995.941: [ParNew: 230230K->8627K(249216K), 0.0107540 secs] 5687617K->5466913K(8360960K), 0.0109030 secs] [Times: user=0.11 sys=0.00, real=0.01 secs]
319050.363: [GC 319050.363: [ParNew: 230195K->9076K(249216K), 0.0118420 secs] 5688481K->5468316K(8360960K), 0.0120470 secs] [Times: user=0.12 sys=0.01, real=0.01 secs]
319134.118: [GC 319134.118: [ParNew: 230644K->8503K(249216K), 0.0105910 secs] 5689884K->5468704K(8360960K), 0.0107430 secs] [Times: user=0.11 sys=0.00, real=0.01 secs]
319159.250: [Full GC (System) 319159.250: [CMS: 5460200K->5412132K(8111744K), 19.1981050 secs] 5497326K->5412132K(8360960K), [CMS Perm : 72243K->72239K(120136K)], 19.1983210 secs] [Times: user=19.14 sys=0.06, real=19.19 secs]
I don’t see the concurrent mark and sweep logs. Does this mean CMS switched to throughput collector? If so why?
Option #2:
Since I see the Full GC (System) logs, I thought of adding -XX\:+DisableExplicitGC. But with that option the collection is not happening and the current heap size is 7.5G. What I am wondering is why CMS is doing the Full GC instead of concurrent collection.

This is a theory ...
I suspect that those CMS collections were not entirely successful. The event at 12477.056 looks like the CMS might have decided that it is not going to be able to work properly due to the 'pre-clean' step taking too long.
If that caused the CMS to decide to switch off, then I expect it will revert to using the classic "throughput" GC algorithm. And there's a good chance it would wait until the heap is full and then it would run a full GC. In short, if you'd just let it continue it would have been OK (modulo that you'd get big GC pauses every now and then.)
I suggest you run your application on a test server with the same heap size and other GC parameters, and see what happens when the server hits the limit. Does it actually throw an OOME?

CMS is running for you :P
You are using incremental mode on CMS (although really you should not bother as its likely punishing your throughput)
The icms_dc in your posted log line is a give away, the only thing that logs this in the JVM is ... the CMS collector, its saying for that GC run you did a small amount of tenure cleanup interwoven with the application.
This part of your log relates to parallel new (the give away there is the heap size)
379022.293: [GC 379022.293: [ParNew: 228000K->4959K(249216K), 0.0152000 secs]
this part is incremenatal CMS (iCMS)
7067945K->6845720K(8360960K) icms_dc=0 , 0.0153940 secs]
I would ask, why are you using iCMS, do you have a lot of Soft/Weak/Phantom references (or why are you using the ParallelRefProcEnabled flag) and have you actually seen an Out of memory, or insufferable pause.
Try backing down to CompressedOops, ParNewGC and CMS without anything else fancy and see if that works out for you.

I can see that the initial heap size -Xms is :8192m and max heap size is -Xmx8192m, which might be one of the reasons why GC is still waiting to start sweeping.
I would suggest to decrease the heap size and then check if the GC kicks in.

When you set the maximum size, it allocates that amount of virtual memory immediately.
When you set the minimum size, it has already allocated the maximum size. All the minimum size does is to take minimal steps to free up memory until this maximum is reached. This could be reducing the number of full GCs because you told it to use up to 8 GB freely.
You have a lot of options turned on (some of them the default) I suggest you strip back to a minimum set as they can have odd interactions when you turn lots of.
I would start with (assuming you have Solaris)
-mx8g -javaagent:lib/instrum.jar -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -verbose\:gc -Xloggc\:/tmp/my-gc.log -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
The options -server is the default on server class machines, -XX:+UseCompressedOops is the default on recent versions of Java and -XX:MaxGCPauseMillis=250 is just a hint.
http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html