I've just tried the -XX:+DoEscapeAnalysis option enabled on a jdk6-u18 VM (on solaris) and had a rather disappointing experience. I'm running a scala application which has rather a lot of actors (20,000 of them). This is a recipe for garbage-creation!
Typically the app can run with 256Mb of heap but generates huge amounts of garbage. In its steady state it:
spends 10% of time in GC
generates >150Mb of garbage in <30s which then gets GC'd
I thought that escape analysis might help so I enabled the option and re-ran the app. I found that the app became increasingly unable to clear away the garbage it had collected until it seemed eventually to spend the entire time doing GC and the app was "flatlining" at its full allocation.
At this point I should say that the app did not throw a OutOfMemoryError which I would have expected. Perhaps JConsole (which I was using to perform the analysis) does not properly display GC stats with this option on (I'm not convinced)?
I then removed the option and restarted and the app became "normal" again! Anyone have any idea what might be going on?
1 Did the escape analysis show up as being enabled in JConsole? You need make sure you're running the VM with the -server option. I assume you had this working, but I just thought I'd check.
2 I don't think escape analysis will help the situation with Scala Actors. You might see a big gain if you do something like:
def act():Unit = {
val omgHugeObject = new OMGHugeObject();
omgHugeObject.doSomethingCrazy();
}
In the example above the EscapeAnalysis would make it so omgHugeObject could be allocated on the stack instead of the heap and thus not create garbage. I don't think it is likely that the escape analysis will help with actors. Their references will always "escape" to the actor subsystem.
3
Are you on the most recent release of Scala? There was a memory leak that I believe was fixed in a recent version. This even caused Lift to spawn off its own Actor library that you might look into.
4 You might try the G1Garbage collector You can enable it with:
-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC
from the jdk-u18 release notes:
Note that Escape analysis-based optimization (-XX:+DoEscapeAnalysis) is disabled in 6u18. This option will be restored in a future Java SE 6 update.
I suggest you try to increase the new generation size, e.g. -XX:NewSize=96M XX:NewRatio=3. Use JVisualVM (included in the JDK), with the Visual GC Plugin to watch how the young and old spaces are utilised.
Related
we have a java8 web application running on tomcat8.5.47 server.we have only 20-60 users sessions per time but most of time up to 600mb uploading files on server.we also use hibernate and c3p0 for manage database connections.
we monitored server several days and saw sometimes java reserved ram increased suddenly and garbage collector did not released it.how can we manage this?and is there any way to release reserved ram and prevent tomcat from increasing ram? and also any way to decrease used ram in task manager?
these are our settings:
-XX:MaxPermSize=1g -XX:+UseG1GC -XX:+UseStringDeduplication -XX:MaxHeapFreeRatio=15 -XX:MinHeapFreeRatio=5 -XX:-UseGCOverheadLimit -Xmn1g -XX:+UseCompressedOops -Xms10g -Xmx56g
and it is an image of profiler when this happened:
and it is an image of profiler and also task manager after 2 hours:
P.s. we use jprofiler to profile and the green colour shows reserved ram and the blue colour is for used ram.also in second box you can track gc activity and third is for classes and forth shows threads activities and last is for cpu activities.
Thank you all for your answers.
These types of questions are never easy, mainly because to get it "right", the person asking them needs to have some basic understanding of how an OS treats and deals with memory; and the fact that there are different types of memory (at least resident, committed and reserved). I am by far not versatile enough to get this entirely right too, but I keep learning and getting better at this. They mean very different things and some of them are usually irrelevant (I find reserved to be such). You are using windows, as such this, imho is a must watch to begin with.
After you watch that, you need to move to the JVM world and how a JVM process. The heap is managed by a garbage collector, so to shrink some un-used heap - the GC needs to be able to do that. And while, before jdk-12, G1 could do that - it was never very eager to. Since jdk-12, there is this JEP that will return memory back, i.e.: it will un-commit memory back. Be sure to read when that happens, though. Also notice that other collectors like Shenandoah and/or ZGC do it much more often.
Of course, since you disable -UseGCOverheadLimit, you get a huge spike in CPU (GC threads are running like crazy to free space) and of course everything slows down. If I were you, i would enable that one back, let GC fail and analyze GC logs to understand what is going on. 56GB of Heap is a huge number for 20-60 users (this surely looks like a leak?). Notice, that without GC logs, this might be impossible to give a solution to.
P.S. Look at the first screen you shared and notice how there are two colors there: green and blue. I don't know what tool is that, but it looks like green is for "reserved memory" and blue is "used" (this is what used means). But it would be great if you said exactly what those are.
Java8 doesn't return allocated RAM back to OS even if JVM doesn't need it. For that feature you need to move to another version of JDK. This is JEP for that https://openjdk.java.net/jeps/346 it says that it was delivered in version 12 so I assume JDKs with version after 12 should have that feature.
The only way to prevent increasing of reserved memory is to decrease Xmx value. And since you are setting it to 56g I assume you are OK with Tomcat consuming up to 56g of memory. So if you think that it is too much then just decrease that number.
I'm running java with java -Xmx240g mypackage.myClass
OS is Ubuntu 12.10.
top says MiB Mem 245743 total, and shows that java process has virt 254g since the very beginning, and res is steadily increasing up to 169g. At that point it looks like it starts garbage collect a lot, I think so because the program is single-threaded at that point, and CPU% is mostly 100% up to this point, and it jumps around 1300-2000 at this point (I conclude it is multithreaded garbage collector), and then res slowly moves to 172g. At that point java crashes with
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at the line with new double[2000][5]
java -version says
java version "1.7.0_15"
OpenJDK Runtime Environment (IcedTea7 2.3.7) (7u15-2.3.7-0ubuntu1~12.10)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
Hardware is Amazon cr1.8xlarge instance
It seems to me that java crashes even when there's a lot of memory available. It is clearly not possible, I have to interpret some numbers wrong. Where should I look to understand what's going on?
Edit:
I don't specify any GC options. The only command-line option is -Xmx240g
My program is successfully working on many inputs, and top said sometimes that it uses up to 98.3% of memory. However I reproduced the situation described above with certain program input.
Edit2:
This is scientific application. It has gigantic tree (1-10 millions of nodes), in each node there are couple double arrays with size approx. 300x3 - 900x5. After initial tree creation program does not allocate much memory. Most of the time there are some arithmetic operations going on with these arrays.
Edit3:
HotSpot JVM died the same way, used CPU a lot at 170-172g mark and crashed with the same error. Looks like 70-75% of memory is some magical line that JVM does not want to cross.
Final solution:
With -XX:+UseConcMarkSweepGC -XX:NewRatio=12 program made it through 170g mark and is happily working further.
Analysis
The first thing you need to do is get a heap dump so you can figure out exactly what the heap looks like when the JVM crashes. Add this set of flags to the command line:
-XX:+HeapDumpOnOutOfMemoryError -verbose:gc -XX:+PrintGCDetails
When a crash happens, the JVM is going to write out the heap to disk. And frankly, its going to take a long time on a heap that size. Download Eclipse MAT or install the plugin if you're already running Eclipse. From there, you can load up the heap dump and run a couple of canned reports. You'll want to check the Leak Suspects and Dominator Tree to see where your memory is going and determine that you don't have an actual leak.
After that, I would recommend you read this document by Oracle about Garbage Collection, however here are some things you can consider:
Concurrent GC
-XX:+UseConcMarkSweepGC
I've never heard of anyone getting away with using the parallel only collector on a heap that size. You can activate the concurrent collector, and you'll want to read up on incremental mode and determine if its right for your workload / hardware combo.
Heap Free Ratio
-XX:MinHeapFreeRatio=25
Dial this down to lower the bar for the garbage collector when you do a full collection. This may prevent you from running out of memory doing a full collection. 40% is the default, experiment with smaller values.
New Ratio
-XX:NewRatio
We'll need to hear more about your actual workload: is this a webapp? A swing app? Depending on how long objects are expected to remain alive on the heap will have an impact on the new ratio value. Server-mode VMs like the one you're running have a fairly high new ratio by default (8:1), this may not be ideal for you if you have a lot of long-lived objects.
If I understood your question correcly, it looks like memory leak actually happening before the program hits the line new double[2000][5]. It seems the memory is already low whe nthe line is hit, thus it throws up when this line asks for more memory.
I would use jvisualvm or similar tools to find out where the memory leak is. Memory leak I've encountered mostly to do with Strings being created in a loop, Cache not being cleared etc.
As a general advice, NEVER use OpenJDK, even less for production environments, it is much slower than the one from Sun/Oracle.
Apart from that I have never seen VM using sooo much memory, but I guess that is what you need (or maybe you have a code using more memory than needed?)
EDIT : OpenJDK for server is fine, only differences with Sun/Oracle JDK is regarding desktop stuff (sound, gui...) so ignore that part.
I'm having trouble with a JVM running an app, whose heap memory looks like a comb. It's constantly jumping from 1.5 GB to 3 GB and slowly deteriorating to higher values. I'm using G1 GC algorithm, but have no idea how to configure it.
I do not have access to the code of the app I'm running and, needless to say, it's a rather large app.
So, bottom line, does anyone know of a good guide to configure GC in Java?
Tuning a JVM is a touch of art and science combined, there's no one answer for everyone. The situation you're seeing sounds like the default GC is doing a "mark now, sweep later" which can lead to pauses during the sweep. One set of options you can try:
JAVA_OPTS="-server -XX:-UseParallelGC -XX:-UseConcMarkSweepGC"
That combo of options basically runs another thread (well 2) in the background which constantly mark AND sweep the GC. The penalty you pay is a bit more CPU use but in this modern day of CPUs it's hardly noticeable for most people.
Here are all the options: http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
You really need to read as many articles on 'java performance tuning' via Google as you can find, you need to profile your application (verbose GC) to find out what it's doing to get the correct combination of params to Java. Like anything else, just blindly throwing out options without knowing what they do can lead to worse performance, not better.
Here it is Tuning Garbage Collection with the 5.0 Java Virtual Machine from Sun...oops Oracle ;-)
Here is the link for Java 6
Currently the most up-to-date resources for GC in a Hotspot VM are the two following:
Java Performance - contains a full chapter dedicated to GC tuning
Java One 2012 Advanced JVM Tuning - most of the talk is dedicated to GC tuning.
My Linux server need to be able to handle 30+ eclipse instances for developers. I did a quick test of running 10 eclipse instances. The Java process associated with each eclipse initially around 200MB RSS memory, increased up to around 550MB, when more projects are loaded.
But Java process doesn't seem to release memory, after closing/deleting all projects within eclipse instances. I still see it uses over 550MB RSS.
How can I change Eclipse or Java settings so that memory foot print got reduced when developers closed down projects or being idle for a while?
Thanks
You may want to experiment with these (and other) JVM tuning options to make the JVM less reluctant to return memory to the OS:
-XX:MaxHeapFreeRatio Maximum percentage of heap free after GC to avoid shrinking. Default is 70.
-XX:MinHeapFreeRatio Minimum percentage of heap free after GC to avoid expansion. Default is 40.
However, I suspect that you won't see the eclipse process shrink to anywhere near its initial size, since eclipse is a huge, complex application that probably lazy-loads (but does not unload, once used) a lot of classes and associated data structures.
I've never seen Java release memory.
I don't think you will get any value out of trying to get it to release memory with Eclipse, I've watched that little memory counter for YEARS and never once see the allocated memory drop.
You might try one of these.
After each session, exit the JVM and restart.
Set your -Xmx lower.
Separate your instances into categories with high -Xmx and low -Xmx and let the user determine which one he wants.
As a side-thought, if it really mattered to you, you MIGHT be able to run multiple eclipse instances under one VM. It would probably be WAY too much work (man-weeks to man-years), but if you could get it right you could reduce overhead by like 150-200mb/instance. The disadvantage would be that a VM crash (Pretty rare these days) would kill everyone.
Testing this theory would be a matter of calling eclipse's main from within an existing JVM and trying to get it to display somewhere useful. The rest of the man-year is spent trying to figure out where they used evil static variables or singletons and changing them to something else.
Switch the Java to use the G1 garbage collector with the HeapFreeRatio parameters. Use these options in eclipse.ini:
-XX:+UnlockExperimentalVMOptions
-XX:+UseG1GC
-XX:MinHeapFreeRatio=5
-XX:MaxHeapFreeRatio=25
Now when Eclipse eats up more than 1 GB of RAM for a complicated operation and switched back to 300 MB after Garbage Collection the memory will be released back to the operating system.
I would suggest checking on garbage collection, setting right options or even forcing GC periodically might increase time till eclipse memory usage grows high.
Following link might be useful http://www.eclipsezone.com/eclipse/forums/t93757.html
I've been tasked with debugging a Java (J2SE) application which after some period of activity begins to throw OutOfMemory exceptions. I am new to Java, but have programming experience. I'm interested in getting your opinions on what a good approach to diagnosing a problem like this might be?
This far I've employed JConsole to get a picture of what's going on. I have a hunch that there are object which are not being released properly and therefor not being cleaned up during garbage collection.
Are there any tools I might use to get a picture of the object ecosystem? Where would you start?
I'd start with a proper Java profiler. JConsole is free, but it's nowhere near as full featured as the ones that cost money. I used JProfiler, and it was well worth the money. See https://stackoverflow.com/questions/14762/please-recommend-a-java-profiler for more options and opinions.
Try the Eclipse Memory Analyzer, or any other tool that can process a java heap dump, and then run your app with the flap that generates a heap dump when you run out of memory.
Then analyze the heap dump and look for suspiciously high object counts.
See this article for more information on the heap dump.
EDIT: Also, please note that your app may just legitimately require more memory than you initially thought. You might try increasing the java minimum and maximum memory allocation to something significantly larger first and see if your application runs indefinitely or simply gets slightly further.
The latest version of the Sun JDK includes VisualVM which is essentially the Netbeans profiler by itself. It works really well.
http://www.yourkit.com/download/index.jsp is the only tool you'll need.
You can take snapshots at (1) app start time, and (2) after running app for N amount of time, then comparing the snapshots to see where memory gets allocated. It will also take a snapshot on OutOfMemoryError so you can compare this snapshot with (1).
For instance, the latest project I had to troubleshoot threw OutOfMemoryError exceptions, and after firing up YourKit I realised that most memory were in fact being allocated to some ehcache "LFU " class, the point being that we specified loads of a certain POJO to be cached in memory, but us not specifying enough -Xms and -Xmx (starting- and max- JVM memory allocation).
I've also used Linux's vmstat e.g. some Linux platforms just don't have enough swap enabled, or don't allocate contiguous blocks of memory, and then there's jstat (bundled with JDK).
UPDATE see https://stackoverflow.com/questions/14762/please-recommend-a-java-profiler
You can also add an "UnhandledExceptionHandler" to your Application's Thread. This will catch 'uncaught' exception, like an out of memory error, and you will at least have an idea where the exception was thrown. Usually this not were the problem is but the 'new' that couldn't be satisfied. As a rule I always add the UnhandledExceptionHandler to a Thread if nothing else to add logging.