I had a tough question during an interview today. They asked me how do I find the root cause of an out of memory error in my production code. I started talking about profilers. The interviewer pulled it to a different direction by saying how do I generate the traffic that's happening in production in order to benefit from the profiler. I didn't have a clue. Any information will be very helpful.
Profilers are not the best tool to chase memory leaks in production
profilers are not run on production JVMs
it is often impossible to recreate in dev or test the conditions that created the memory leak in production (the interviewer hint)
The solution is to take a heap dump (using jmap for instance - JVM dependent) and analyze it using a tool like the excellent (and freely available) SAP Memory Analyzer (MAT)
Related
I am suspecting a native memory leak in my java code. Are there any tools which do native memory profiling? Also, does any tool support native memory analysis of a running java process?
Thanks!!
Edit:
I have already tried Memory Validator and Purify but it seems they support only 32-bit processes. Is there some tool similar to the above ones which can simply attach to a running windows process and give us native memory analysis for that particular process?
The Troubleshooting guide for Java SE 6 with Hotspot VM contains a fairly elaborate section on techniques to aid in detecting native memory leaks. These include:
wrapping all memory allocation and deallocation calls to track the amount of memory used.
relying on platform specific support like the debug support provided by the Microsoft Visual C++ compiler or on mtrace (and MALLOC_TRACE) to debug memory allocations on Linux.
using memory leak analysis tools like Rational Purify.
among others. Notably, the article mentions that no ideal solution exists for all platforms.
Also, consider using the -Xcheck:jni flag that appears to be available in most JVMs. The -X flag itself indicates that the flag is non-standard, but the flag appears to be available in the IBM JDK, Oracle JRockit R28, and even the Oracle/Sun JVM. Enabling the flag switches on a mode where wrappers are added around JNI calls, thereby allowing you to track illegal arguments passed to JVM calls as noted in the JNI programmers' guide and specification. While, it's use in detecting memory leaks is subjective, it would definitely help if you suspect that the leak is being caused due to invalid parameters being issued.
AFAIK you can't do it with Java tools like JProfiler, JVisualVM etc. If you have memory leak in native code use tools for native code. You ie. can run it from C (i.e. loading jvm.dll). You can look at this articles finding memory leaks using Visual Studio or Memory Leak Detection in C++ (Linux)
Note: of course if you leak is connected to heap leak (forgot about deleteglobalref) you can find it with Java tools, but it is very rare in JNI.
I have been working on an open-source project named "MySafe" (https://github.com/serkan-ozal/mysafe) It basicly intercepts and monitors "Unsafe" calls. (In fact, it makes more than). With version 2.0, it can be useful for tracking and detecting "Unsafe" based native memory leaks.
Demo code: https://github.com/serkan-ozal/mysafe/blob/master/src/test/java/tr/com/serkanozal/mysafe/NativeMemoryLeakHuntingDemo.java
Diagram showing source of the leak: https://github.com/serkan-ozal/mysafe/blob/master/src/test/resources/native-memory-leak-hunting.png
To diagnose native memory leak, JIT code symbol mapping and Linux recent profiling tools are required: perf, perf-map-agent and bcc.
Please refer to details in related answer https://stackoverflow.com/a/52767721/737790
Many thanks to Brendan Gregg
These are tools you can use for debugging
libtcmalloc HPROF: For heap profiling
The jcmd Utility, PSS for the process: Can help in confirming a native leak.
Native Memory Tracking: Tracking native memory leak in JVM (only works for allocation inside JVM)
Core dump analysis, pmap and gdb checking for anon blocks and process memory overtime
-Xcheck:jni
Further details can be found out here
https://www.bro-code.in/blog/java-debugging-native-memory-leak
http://www.oracle.com/technetwork/java/javase/memleaks-137499.html#gbyvk
I'm a big fan of JProfiler. That's the best tool for profiling and memory leaks. It's fairly cheap relative to most tools, really easy to learn, and lots of features.
http://www.ej-technologies.com/products/jprofiler/overview.html
I'm trying to determine whether or not I have a memory leak in my webapp. I'm using VisualVM and JMeter to load test and watch the heap.
I saved a heap dump to file and downloaded Eclipse Memory Analyzer yesterday...after much frustration with VisualVM, I thought Eclipse would pinpoint the leak, if any, better than VisualVM.
I opened the heap file in Eclipse and ran what they call a Leak Suspects Report. I thought it would point to a particular class in my webapp, but it doesn't. So I have no clue how to use the info its provided in order to find out where in any particular class of mine the leak suspect is.
Here's the results of the Leak Suspect Report for one of my heap dump files.
One instance of "org.apache.catalina.session.StandardManager" loaded by "org.apache.catalina.loader.StandardClassLoader # 0x261bdac0" occupies 16,977,376 (48.54%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "".
Keywords
org.apache.catalina.loader.StandardClassLoader # 0x261bdac0
org.apache.catalina.session.StandardManager
java.util.concurrent.ConcurrentHashMap$Segment[]
The rest of the Details in the report are as shown in the attached image. I hope the image can be expanded for a closer look....
I know that Eclipse is supposed to be really good software. This is my last attempt to use something like this to find a memory leak - I just have very, very, limited knowledge in HOW this software can be used for such. Tutorial and help pages describe things as though you should know what to do after a few clicks... I need more help than that.
While I don't have any experience with using Eclipse for finding leaks, I would ask a question first: How sure are you that you have a memory leak? From your question, it doesn't sound like you are sure you have a leak, but you are testing to see if you do have one. The simplest way to test that would be to start your application, note how much memory it is consuming, have JMeter hit it continuously for 24 hours, and see how much memory it is consuming (probably after executing GC). If your application is consuming a significantly large portion of memory, or has died from an OutOfMemoryError, then you have a memory leak.
If you find that you actually do have a memory leak, then I would first suggest running your application through FindBugs to see if it can find the memory leaks through a quick static analysis. If that doesn't work, then this article (although it is rather old) might help you understand the results given to you by Eclipse.
Is there any difference between Java Memory Analysis Tools (profilers like Yourkit Java Profiler) and Java Memory Leak Detection Tools? (From my searches, the difference is pretty hard to notice in what tools are concerned, but it sounds to me like a difference in the approach manner).
If there is, are there any free (available for download) tools for memory leak detection (not memory profiling)? Thank you in advance!
Strictly speaking, there are no memory leaks in java in the C/C++ sense: memory allocation with no more reference to it.
So when the term "leak" is used in the java context, sometimes it just means a big memory structure. Some tools call them "leak suspects", because they cannot judge if a memory structure is big due to the nature of the problem or by mistake.
And sometimes "leak" is used in the dynamic sense: a memory structure that is growing over time.
So in the first place, you need a memory tool that gives you a quantitive overview on memory distribution and referer structure in a java heap dump snapshot. Having that, you can investigate the dynamic behaviour by comparing two or more snapshots.
If these tools offer an addional "Leak-Feature" (JProbe's Memory-Leak-Doctor, Eclipse-MATs Memory-Leak Report) it indicates that they have problems presenting a really comprehensive view on memory structures in the first place.
If you are looking for a free download, look at the
http://eclipse.org/mat/
and at
http://dr-brenschede.de/bheapsampler/
Think of Memory Leak Detection Tools as just an optimized version of a memory analysis tool that helps you find a problem without having to go scouring through the data yourself.
I've used Yourkit. I would argue it has both features. While it doesn't explicitly point out "yo dude, your leak is on this line of code" it provides utility for comparing snapshots of memory to show what changed over time and provides various utilities for chasing the references of objects to figure out why they are still being held in memory.
Depending on your definition of leak detection tools, one could argue that the graph in JConsole that shows heap memory usage is a leak detection tool. Look at a graph of heap consumption over a long enough period of usage...if the average value keeps going up, good chance you have a leak.
I'm not related to the Yourkit folks, but I can recommend the tool. It could be better, but it isn't bad. I use it for both memory and cpu performance analysis. I do regular load and memory testing (JConsole/JMX and other methods) on our product and use Yourkit when something seems odd. About once every 2-4 months I do a cpu analysis looking for easy to fix performance bottlenecks (our load testing catches major problems).
I have a standalone program that I run locally, it is meant to be a server type program running 24/7. Recently I found that it has a memory leak, right now our only solution is to restart it every 4 hours. What is the best way to go about finding this memory leak? Which tool and method should we use?
If you are using Java from Sun and you use at least Java 6 update 10 (i.e. the newest), then try running jvisualvm from the JDK on the same machine as your program is running, and attach to it and enable profiling.
This is most likely the simplest way to get started.
When it comes to hunting memory problems, I use SAP Memory Analyzer Eclipse Memory Analyser (MAT), a Heap Dump analysis tool.
The Memory Analyzer provides a general purpose toolkit to analyze Java heap dumps. Besides heap walking and fast calculation of retained sizes, the Eclipse tool reports leak suspects and memory consumption anti-patterns. The main area of application are Out Of Memory Errors and high memory consumption.
Initiated by SAP, the project has since been open sourced and is now know as Eclipse Memory Analyser. Check out the Getting Started page and especially the Finding Memory Leaks section (I'm pasting it below because I fixed some links):
Start by running the leak report to automatically check for memory leaks.
This blog details How to Find a Leaking Workbench Window.
The Memory Analyzer grew up at SAP. Back then, Krum blogged about Finding Memory Leaks with SAP Memory Analyzer. The content is still relevant!
This is probably the best tool you can get (even for money) for heap dump analysis (and memory leaks).
PS: I do not work for SAP/IBM/Eclipse, I'm just a very happy MAT user with positive feedback.
You need a memory profiler. I recommend trying the Netbeans profiler.
One approach would be to take heap dumps on a regular basis, then trend the instance counts of your classes to try to work out which objects are being consistently created but not collected.
Another would be to switch off parts of your app to try to narrow down where the problem is.
Look at tools like jmap and jhat.
You might look up JMX and the jconsole app that ships with Java. You can get some interesting statistics out-of-the-box, and adding some simple instrumentation to your classes can provide a whole lot more.
As already stated jvisualvm is a great way to get started, but once you know what is leaking you may need to find what is holding references to the objects in question for which I'd recommend jmap and jhat, e.g
jmap -dump:live,file=heap.dump.out,format=b <pid>
and
jhat heap.dump.out
where <pid> is easily found from jvisualvm. Then in a browser navigate to localhost:7000 and begin exploring.
You need to try and capture Java heap dump which is a memory print of the Java process.
It's a critical process for memory consumption optimisation and finding memory leaks.
Java heap dump is an essential object for diagnosing memory-linked issues including java.lang.OutOfMemoryError, Garbage Collection issues, and memory leaks which are all part of Java web development process
For clarity, a Heap dump contains information such as Java classes and objects in a heap during instant of taking the snapshot.
To do it, you need to run jmap -dump:file=myheap.bin <program pid>.
To learn more about how to capture Java heat dumps, check out: https://javatutorial.net/capture-java-heap-dump
I'm working on a large application (300K LOC) that is causing a memory leak in the Sun 1.6 JVM (1.6_05). Profiling the Java shows no leak. Are there any diagnostics available from the JVM that might detect the cause of the leak?
I haven't been able to create a simple, isolated Java test case. Is the only way to figure this out by using a C heap analyzer on the JVM?
The application creates a pool of sockets and does a significant amount of network I/O.
Some profiler like profiler4j can show the managed and the unmanaged memory (live curve). Then you can see if you has a leak and when the leak occur. But you does not find more informations.
After this there are 2 possible solutions:
You can with the live curve isolate the problem and create a simpler test until you have find the cause of the problem.
You search your code for the typical problems like:
Instances of the class Thread that are never start.
Images or Graphics that never are dispose
ODBC Bridge Objects that are never close
I love valgrind ( http://valgrind.org/ ), if you are executing it on a system it supports. It really rocks!