I am suspecting a native memory leak in my java code. Are there any tools which do native memory profiling? Also, does any tool support native memory analysis of a running java process?
Thanks!!
Edit:
I have already tried Memory Validator and Purify but it seems they support only 32-bit processes. Is there some tool similar to the above ones which can simply attach to a running windows process and give us native memory analysis for that particular process?
The Troubleshooting guide for Java SE 6 with Hotspot VM contains a fairly elaborate section on techniques to aid in detecting native memory leaks. These include:
wrapping all memory allocation and deallocation calls to track the amount of memory used.
relying on platform specific support like the debug support provided by the Microsoft Visual C++ compiler or on mtrace (and MALLOC_TRACE) to debug memory allocations on Linux.
using memory leak analysis tools like Rational Purify.
among others. Notably, the article mentions that no ideal solution exists for all platforms.
Also, consider using the -Xcheck:jni flag that appears to be available in most JVMs. The -X flag itself indicates that the flag is non-standard, but the flag appears to be available in the IBM JDK, Oracle JRockit R28, and even the Oracle/Sun JVM. Enabling the flag switches on a mode where wrappers are added around JNI calls, thereby allowing you to track illegal arguments passed to JVM calls as noted in the JNI programmers' guide and specification. While, it's use in detecting memory leaks is subjective, it would definitely help if you suspect that the leak is being caused due to invalid parameters being issued.
AFAIK you can't do it with Java tools like JProfiler, JVisualVM etc. If you have memory leak in native code use tools for native code. You ie. can run it from C (i.e. loading jvm.dll). You can look at this articles finding memory leaks using Visual Studio or Memory Leak Detection in C++ (Linux)
Note: of course if you leak is connected to heap leak (forgot about deleteglobalref) you can find it with Java tools, but it is very rare in JNI.
I have been working on an open-source project named "MySafe" (https://github.com/serkan-ozal/mysafe) It basicly intercepts and monitors "Unsafe" calls. (In fact, it makes more than). With version 2.0, it can be useful for tracking and detecting "Unsafe" based native memory leaks.
Demo code: https://github.com/serkan-ozal/mysafe/blob/master/src/test/java/tr/com/serkanozal/mysafe/NativeMemoryLeakHuntingDemo.java
Diagram showing source of the leak: https://github.com/serkan-ozal/mysafe/blob/master/src/test/resources/native-memory-leak-hunting.png
To diagnose native memory leak, JIT code symbol mapping and Linux recent profiling tools are required: perf, perf-map-agent and bcc.
Please refer to details in related answer https://stackoverflow.com/a/52767721/737790
Many thanks to Brendan Gregg
These are tools you can use for debugging
libtcmalloc HPROF: For heap profiling
The jcmd Utility, PSS for the process: Can help in confirming a native leak.
Native Memory Tracking: Tracking native memory leak in JVM (only works for allocation inside JVM)
Core dump analysis, pmap and gdb checking for anon blocks and process memory overtime
-Xcheck:jni
Further details can be found out here
https://www.bro-code.in/blog/java-debugging-native-memory-leak
http://www.oracle.com/technetwork/java/javase/memleaks-137499.html#gbyvk
I'm a big fan of JProfiler. That's the best tool for profiling and memory leaks. It's fairly cheap relative to most tools, really easy to learn, and lots of features.
http://www.ej-technologies.com/products/jprofiler/overview.html
Related
JVM native memory leak is suspected, so I want to track memory.
I tried Native tracking using the JVM tuning flag as follows, but I couldn't find a memory leak point.
java -XX:NativeMemoryTracking=detail
Probably because of the following reasons in the document.
NMT in this release does not track third party native code memory allocations and JDK class libraries.
Also, this release does not include NMT MBean in HotSpot for JMC.
Is there any other way to track JVM native memory?
There are several tools that you can use.
I'll refer to this excellent answer by apangin: Java using much more memory than heap size (or size correctly Docker memory limit)
I strongly encourage you to read all of that but for your question, this in particular is relevant:
There are tools and techniques for investigating issues with Java memory consumption: Native Memory Tracking, pmap, jemalloc, async-profiler.
I have project written in java that using JNI uses a C++ library.
All the code was written by us, so I have all the source code.
After few hours the machine runs out of memory although my process just iterate over files and all the memory regarding the previous file deleted.
I'm sure that there is a memory leak, usually I use Valgrind, but it seems he can't cope with Java very well and believes that the JVM is leaking, even for "hello world" java project.
I've tested the C++ parts (the major flows) with unit-tests and used valgrind on the unit-tests, but couldn't find any leakage. It doesn't prove anything because there are many potential flows I could've missed.
My major question, how can I find my leak ?
It will be extremely helpful to know who is consuming the memory, the java or the native part ? they are in the same process.
Thanks.
You can use jemalloc to debug native memory leaks. This blog post has a detailed example of using jemalloc to debug a native memory leak in java applications.
From my experience, Valgrind is actually usable with the JVM, and it remains the best tool to hunt leaks in C/C++ code, even with JNI. So your question kinda contains the answer that you need ;)
If you fail to use JNI and Valgrind together, please refer to Valgrind and Java.
I had a tough question during an interview today. They asked me how do I find the root cause of an out of memory error in my production code. I started talking about profilers. The interviewer pulled it to a different direction by saying how do I generate the traffic that's happening in production in order to benefit from the profiler. I didn't have a clue. Any information will be very helpful.
Profilers are not the best tool to chase memory leaks in production
profilers are not run on production JVMs
it is often impossible to recreate in dev or test the conditions that created the memory leak in production (the interviewer hint)
The solution is to take a heap dump (using jmap for instance - JVM dependent) and analyze it using a tool like the excellent (and freely available) SAP Memory Analyzer (MAT)
I have a standalone program that I run locally, it is meant to be a server type program running 24/7. Recently I found that it has a memory leak, right now our only solution is to restart it every 4 hours. What is the best way to go about finding this memory leak? Which tool and method should we use?
If you are using Java from Sun and you use at least Java 6 update 10 (i.e. the newest), then try running jvisualvm from the JDK on the same machine as your program is running, and attach to it and enable profiling.
This is most likely the simplest way to get started.
When it comes to hunting memory problems, I use SAP Memory Analyzer Eclipse Memory Analyser (MAT), a Heap Dump analysis tool.
The Memory Analyzer provides a general purpose toolkit to analyze Java heap dumps. Besides heap walking and fast calculation of retained sizes, the Eclipse tool reports leak suspects and memory consumption anti-patterns. The main area of application are Out Of Memory Errors and high memory consumption.
Initiated by SAP, the project has since been open sourced and is now know as Eclipse Memory Analyser. Check out the Getting Started page and especially the Finding Memory Leaks section (I'm pasting it below because I fixed some links):
Start by running the leak report to automatically check for memory leaks.
This blog details How to Find a Leaking Workbench Window.
The Memory Analyzer grew up at SAP. Back then, Krum blogged about Finding Memory Leaks with SAP Memory Analyzer. The content is still relevant!
This is probably the best tool you can get (even for money) for heap dump analysis (and memory leaks).
PS: I do not work for SAP/IBM/Eclipse, I'm just a very happy MAT user with positive feedback.
You need a memory profiler. I recommend trying the Netbeans profiler.
One approach would be to take heap dumps on a regular basis, then trend the instance counts of your classes to try to work out which objects are being consistently created but not collected.
Another would be to switch off parts of your app to try to narrow down where the problem is.
Look at tools like jmap and jhat.
You might look up JMX and the jconsole app that ships with Java. You can get some interesting statistics out-of-the-box, and adding some simple instrumentation to your classes can provide a whole lot more.
As already stated jvisualvm is a great way to get started, but once you know what is leaking you may need to find what is holding references to the objects in question for which I'd recommend jmap and jhat, e.g
jmap -dump:live,file=heap.dump.out,format=b <pid>
and
jhat heap.dump.out
where <pid> is easily found from jvisualvm. Then in a browser navigate to localhost:7000 and begin exploring.
You need to try and capture Java heap dump which is a memory print of the Java process.
It's a critical process for memory consumption optimisation and finding memory leaks.
Java heap dump is an essential object for diagnosing memory-linked issues including java.lang.OutOfMemoryError, Garbage Collection issues, and memory leaks which are all part of Java web development process
For clarity, a Heap dump contains information such as Java classes and objects in a heap during instant of taking the snapshot.
To do it, you need to run jmap -dump:file=myheap.bin <program pid>.
To learn more about how to capture Java heat dumps, check out: https://javatutorial.net/capture-java-heap-dump
The sun JVM spits out a LOT of extra noise when run under valgrind, which makes tracking memory problems in the application very challenging.
I'd like to find either a suppression file, or a VM runtime mode, that will strip out spurious memory errors in order to separate the wheat from the chaff in this situation. Any suggestions?
What about profiling this native code outside of the Java application? Usually the JNI code is a wrapper around some library that is not Java specific. Not sure if that is true for your specific case, but if it is then maybe the memory problems can be isolated by writing a plain C or C++ test framework around that library?
If your framework is in C++ then you might also be able to supply your own new and delete operators and track memory usage yourself. You'll have to collect statistics and process them with some scripts but it can work well.
I can't answer your posted question, but can you elaborate on what problem you're having?
In other words, can you tell us if it is...
In the JNI layer and not a JVM object scope issue?
A use of free'd memory?
A buffer underwrite/overwrite?
Other memory corruption?
I recently had to debug a Java/C that had issues (after 30+ minutes into its run), which it turns out was using memory after it had been free'd. I tried using dmalloc, a custom memory leak library of mine, Valgrind and none worked as I needed.
Eventually I created a simple set of wrappers around free, malloc, calloc, realloc that simply printed memory addresses and sizes to a file. After it aborted (within GDB) I could backtrack in time and figure out when the memory was free'd and where the references were that did not get removed.
IF your issue is in C/C++ and you can trap the error in a debugger this might work for you. Yes, it's tedious, but maybe no worse than sifting through megabytes of Valgrind output.
Hope that helps & good luck.
While not as spiffy as valgrind (based on what I've read), you might try jmap and jhat. These tools allow you to take a snapshot of the running process and see what's going on. I've used this technique for simple memory leaks and it's worked well. However, if the memory problems are caused by non-jvm allocations, this won't help.