I have an application deployed to production which sometimes throws OutOfMemory exception due to some memory leak. It is running on a headless ubuntu box on which I would prefer not to connect visualvm, jconsole etc remotely. Is there a way to make the jvm do gc (like in visualvm where you just click a button to do it).
I would like to run jmap -histo:live <pid> and this gc commands alternatively to find out which objects are surviving a gc. Which object numbers are growing etc. Right now I can see some unexpected object counts but it is happening across a number of my domain objects so I am not sure if it is a delayed gc or a memory leak.
So in short, I am looking for the linux command to be run against a jvm pid to cause it to do gc. Not system.gc.
The GC will aggressively try to clean up unreferenced objects as the heap gets full. So its not a "delayed gc". I think you are on the right track, use jmap and get a heap dump. Then analyze it to see what application objects are surviving that should not be. You may need to get a couple heap dumps and compare them against each other.
It's pretty hard to get a real memory leak in Java. If you're getting an out of memory error, then it most likely means that you're actually running out of memory. So to fix this, you need to find references to unused objects and clean them up manually. Because otherwise, the garbage collector can't free up the wasted memory.
When the JVM can't allocate any more memory, the garbage collector should automatically run.
Related
I have a java.lang.OutOfMemoryError:GC Overhead limit exceeded.
There is no HeapDumpOnOutOfMemoryError command line option for my application.
I need a heap dump but when I try to capture the dump with jmap or jcmd tools they are not responding:
jmap
D:\program>jmap -dump:live,format=b,file=d:\dump4.hprof 8280
Dumping heap to D:\dump4.hprof ...
jcmd
D:\program>jcmd 8280 GC.heap_dump d:\dump6.hprof
8280:
Processes are not completing but dump files are created. When I open them with VisualVM, they are loading infinitely.
If I capture a heap dump of e.g. VisualVM, Tools complete successfully and dumps are created and opened.
Could you please explain why jmap and jcmd are not completing? And how can I capture a dump of the application with OutOfMemoryError exception? Application is still working but there are only a few live threads.
One possibility is that the heap size you intend to dump is too large in size.
Please specify the size of the heap and RAM.
It is not due to your intended heap size is more than allocated heap size. This error occurs when the JVM spent too much time performing Garbage Collection and was only able to reclaim very little heap space. Your application might have ended up using almost all the RAM and Garbage collector has spent too much time trying to clean it and failed repeatedly.
Your application's performance will be slow comparatively, This is because the CPU is using its entire capacity for Garbage Collection and hence cannot perform any other tasks.
Following questions need to be addressed:
What are the objects in the application that occupy large portions of the heap?
In which parts of the source code are these objects being allocated?
You can also use automated graphical tools such as JConsole which helps to detect performance problems in the code including java.lang.OutOfMemoryErrors.
My WebLogic server was configured with 16gb of heap space, but it was 90% used within 1 hour of production usage when most of the users started work. I observed there were several stuck threads whenever this happens.
I have captured the heap dump when the heap was approx 10% free. How do I inspect the heap dump to find out the memory leak, or process, codes which is causing this issue.
I have tried to understand the memory leak, running tools like JMap and Eclipse MAT, but it maybe due to lack of experience, I couldn't understand what these tools are trying to show. Or how/what should I look out for?
I have both the before/after GC heap dump to analyze.
I have reviewed the thread dumps, there were no "waiting to lock" objects threads, the threads were similar as shown below, with threads stuck with no obvious reasons.
According to your heap dump, your biggest memory issue is the int arrays, indeed it takes nearly 70 % of your heap (Yes sort the Size Column instead).
Select it in your heap dump, right click and select on Show in Instances View
Then browse the biggest objects and for each of them right click and select Show Nearest GC Root to see which Object has still an hard reference to the int array which prevents to be eligible for the GC.
It could help you to find your memory leak assuming that it is a memory leak.
See below an example of Nearest GC Root allowing to identify a leak that I added intentionally to my program just to show the idea. As you can see in the screenshot, I have an array of int which cannot be eligible for the GC because it is stored in an HashMap called leak in my class Application, so I know that my memory issue could be due to this particular HashMap especially if I have many other objects which lead to this HashMap.
NB: Be patient when you try to identify a leak as it is not always obvious, the ideal situation is where you have a huge object that takes the whole heap but obviously it is not your case there is nothing really obvious that is the reason why I propose to investigate the int arrays first. Don't forget that it could also be little int arrays but thousands of them with the same Nearest GC Root.
Another trick, If you have JProfiler you can simply follow this wonderful tutorial to find your leak.
Response Update:
One simple way to better identify the root cause of the memory leak is to take at least 2 heap dumps then compare them using a tool like jhat with the syntax
jhat -J-Xmx2G -baseline ${path-to-the-first-heap-dump} ${path-to-the-second-heap-dump}
It will launch a small HTTP sever on port 7000 so:
Launch http://localhost:7000/
Then click on Show instance counts for all classes (including platform)
You will then see the list of Classes ordered by total amount of new instances created. You can then use VisualVM to do what I described in the first part of my answer to find the root cause of your memory leak.
You can also use jhat
By selecting of the Top Classes then for each of them
click on one "Reference to this Object"
then click on Exclude weak refs
You will then see the GC root of each instances like the next screenshot:
Another way is to use Eclipse Memory Analyzer also called MAT.
Open the second snapshot with it
Select the view histogram
Then for each of the Top Classes right click
Choose Merge Shortest Paths To GC Roots/ Exclude All references
you will then see something like the next screenshot:
The JDK's "jmap -histo" command will dump object counts/bytes for all classes to a text file. If you capture/compare a few of these dumps over time, you will see which ones grow continually -- your memory leak. The overhead of -histo is much lower than that of capturing a full heap dump.
Comparing just a few dumps (like the python script detailed here) seems like too small of a sample, so I wrote an open-source tool (here) that runs this jmap -histo command in the background (at an interval). It has a live display and tracks the % of time that the byte count for each class is on the rise.
It seems you, probably, have a memory leak situation. Your best approach is to use Java Mission Control with Flight Recorder to get the class and method leaking.
You should setup your weblogic managed server with the following parameters:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8999
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-XX:+UnlockCommercialFeatures
-XX:+FlightRecorder
When you set this up, follow the instructions here to detect the leak.
Hope it helps !!
I am one of the developers of the tool called Plumbr. Among other things we make an automatic analysis of heap contents in case of excessive memory usage. You may find it useful.
Per your comments: you have Java 7 with 16GB heap, no GC algorithm explicitly specified, so default for Java 7 is Throughput GC, which is not suitable for most web apps, for it leads to long GC pauses for big heaps.
Switch to ConcurrentMarkSweep GC, this way GC will not wait till your memory fills up and will try its best to collect garbage incrementally, so that you will have fewer Stop The World pauses.
Did you try yourkit profiler? It's not free, but you can evaluate it for 30 days. In this case if you dump contains all object (not only live), you will be able to check roots for them as well. Because it could be that you don't have memory leak, but too big memory footprint. Also it would be great to enable GC logs and parse how much FullGC pauses do you have:
grep "Full GC" jvm_gc.log | wc -l
In ideal world it should be 0 :)
Btw, whole this article could be helpful for you.
Problem
We are trying to find the culprit of a big memory leak in our web application. We have pretty limited experience with finding a memory leak, but we found out how to make a java heap dump using jmap and analyze it in Eclipse MAT.
However, with our application using 56/60GB memory, the heap dump is only 16GB in size and is even less in Eclipse MAT.
Context
Our server uses Wildfly 8.2.0 on Ubuntu 14.04 for our java application, whose process uses 95% of the available memory. When making the dump, our buffers/cache used space was at 56GB.
We used the following command to create the dump: sudo -u {application user} jmap -dump:file=/mnt/heapdump/dump_prd.bin {pid}
The heap dump file size is 16,4GB and when analyzing it with Eclipse MAT, it says there are around 1GB live objects and ~14,8GB unreachable/shallow heap.
EDIT: Here is some more info about the problem we see happening. We monitor our memory usage, and we see it grow and grow, until there is ~300mb free memory left. Then it stays around that amount of memory, until the process crashes, unfortunately without error in the application log.
This makes us assume it is a hard OOM error because this only happens when the memory is near-depleted. We use the settings -Xms25000m -Xmx40000m for our JVM.
Question
Basically, we are wondering why the majority of our memory isn't captured in this dump. The top retained size classes don't look too suspicious, so we are wondering if there is something heap dump-related what we are doing wrong.
When dumping its heap, the JVM will first run a garbage collection cycle to free any unreachable objects.
How can I take a heap dump on Java 5 without garbage collecting first?
In my experience, in a true OutOfMemoryError where your application is simply demanding more heap space than is available, this GC is a fool's errand and the final heap dump will be the size of the max. heap size.
When the heap dump is much smaller, that means the system was not truly out of memory, but perhaps had memory pressure. For example, there is the java.lang.OutOfMemoryError: GC overhead limit exceeded error, which means that the JVM may have been able to free enough memory to service some new allocation request, but it had to spend too much time collecting garbage.
It's also possible that you don't have a memory problem. What makes you think you do? You didn't mention anything about heap usage or an OutOfMemoryError. You've only mentioned the JVM's memory footprint on the operating system.
In my experience, having a heap dump much smaller than the real memory used can be due to a leak in the JNI.
Despite you don't use directly any native code, there are certain libraries that use it to speed up.
In our case, it was a Deflater and Inflater not properly ended.
I have a Scala daemon application, that runs in a server in Rackspace with a limit of 2GB. Because of an unknown reason, the server get stuck after some time the application is running. I am suspecting there is a memory leak, because the server memory gets full after some time.
I tried to run jvisualvm, making snapshots of memory in two different moments and compare them to see if there were objects that remained allocated, but I could not find anything.
The heap allocation is just around 400MB. Here is a snapshot of the JVM memory in New Relic:
Notice that PS Eden Space heap is what keeps increasing. I did a work around that kills the application every 3 hours and starts it back again (this is why the graph suddenly goes back down).
Any idea of why this PS Eden is increasing? How to fix it?
Edit 1:
Screenshot of the machine that halted minutes before 13:00
Edit 2:
On a new round, a left the server hang itself, and used G1GC. Here is the new relic graph for this run:
It's normal that the Eden grows constantly, that is where new objects are allocated. Eden will keep growing until it get's full or until a partial collection runs that collects unused objects, and shifts objects being used to the survivor region S0.
This is as per the way this type of garbage collection was designed. The idea is that it's OK that Eden is full, we let it grow and garbage collect it only when it's most convenient, minimizing the impact for application code.
Try to remove the workaround, let the server freeze and see if there are any out of memory errors in the logs. Too many classes would have cause such errors.
Try to see if the OldGen is full. Then using visualvm, force a garbage collection, and see if it goes down. If it doesn't, there is the problem.
Then take a heap dump and a thread dump and analyse the heap dump in MAT - Eclipse Memory Analyser tool, see this tutorial as well. it could be that the server just needs more memory.
One important notion, in Java there is really no notion of a memory leak, the garbage collector works mostly flawlessly to collect unused objects.
Usually the problem comes from objects that are created but are for example kept around in static collections or thread local variables accidentally, and because they are referenced get never collected.
A tool that has a free trial and allows to generate a report that pinpoints a lot of these common causes is Plumbr. That is probably the best chance at a quick solution, try to run plumbr to see if it finds something, if not then MAT analisys of the heap dump.
I am looking into how to use JConsole to detect memory leaks.
I see that in Memory Pool in my MBeans I can define UsageThreashold for my Tenured Generation.
So if my application exceeds this threashold the heap memory becomes red in the Memory tab.
Question: How does this help? I mean how am I supposed to use this setting to analyze my memory? How am I supposed to figure out this value?
In my opinion I don't think that UsageThreashold parameter is the most helpful for you to detect memory leaks (but if someone knows some tricks with it, please do share). In my experience that parameter is more helpful to visually understand if my application is getting way too near my max heap size and I'm in danger of getting an OutOfMemoryException.
Still regarding using JConsole to search for memory leaks, I don't think there's a silver bullet for the process. But what I usually do is the following:
If exists a memory leak, it means that the objects (the ones that are leaking) won't get collected, hence, your Tenured Generation won't fully recover after any amount of GCs.
With the application running I connect JConsole and try to spot a leak by observing the memory tab, if after several computations of my application and also after various GCs occurring (including pressing the Perform GC button, which will result in a full gc) the memory never goes below, or at least to the memory value, it started tracking there's a great possibility that something is leaking. When the leak is big, you can even see a "stair graph" pattern in your memory.
Keep in mind that if your application has long computations running, which may consume memory this analyzes must be done carefully. You must understand when those processes have finished. For example, just run one of those computations and track the total evolution of memory, before, during and afterwards.
Also, I suggest you to try visualVM instead, because it also allows you to create heap dumps, which you can use in order to understand which objects are still in memory and explore the references graph to understand why they are not being collected.
you can use JMAP to see the histogram and/or to create heap dumps and study your memory consumption with tools like Eclipse MAT or YourKit.
JConsole is used more for monitoring and running MBeans and less for analysis and in my expirence JVisualvm is better for that since you can use it for sampling your code and see what methods are CPU consuming.