JVM native memory leak is suspected, so I want to track memory.
I tried Native tracking using the JVM tuning flag as follows, but I couldn't find a memory leak point.
java -XX:NativeMemoryTracking=detail
Probably because of the following reasons in the document.
NMT in this release does not track third party native code memory allocations and JDK class libraries.
Also, this release does not include NMT MBean in HotSpot for JMC.
Is there any other way to track JVM native memory?
There are several tools that you can use.
I'll refer to this excellent answer by apangin: Java using much more memory than heap size (or size correctly Docker memory limit)
I strongly encourage you to read all of that but for your question, this in particular is relevant:
There are tools and techniques for investigating issues with Java memory consumption: Native Memory Tracking, pmap, jemalloc, async-profiler.
Related
We're running against an issue related with memory.
Our monitor is saying that out service is consuming 800MiB of memory.
We've created a heap dump. However, this heap dump is telling us that total heap size is 41MiB.
Out service is running inside a container. We've read a lot some issue related with java and docker containers. However, we're trying to get some more information.
How could we otain more memory related information about java vm?
Any ideas?
The big discrepancy between the memory usage reported by the heap dump and the usage reported by external monitoring could be due to a few things. Some things to look at include:
A JVM's memory utilization does not just consist of the Java heap. It includes the JVM executable and shared libraries, metaspace, the native (malloc) heap and memory segments that hold thread stacks. It potentially also includes memory mapped files, and memory segments shared with other processes.
The memory used by the Java heap includes objects that are unreachable, and space that isn't currently allocated to objects; i.e. free heap space.
How can you get more information about Java memory utilization?
A primary source is "Java Platform, Standard Edition Troubleshooting Guide":
The Native Memory Tracking section explains how to use jcmd to get an overall picture.
The JFR Memory Tab provides live monitoring of Java Heap usage stats and the GC.
The Jmap Utility can be used to obtain heap and GC configuration information, heap histograms and classloader stats.
GC logging can give useful insights into GC behavior.
We developed an highly CPU intensive Java server application that is having a serious memory leak (or so it seems). As time passes, the application seems to eat up increasingly more memory (as seen with Windows Task Manager) but if I analyse it a specialized Java profiler the memory seems to be staying the same. For example, in task manager I see the application taking over 8gb of memory and growing, but in the Java profiler I see that heap memory is at most 2gb. I tried all possible combinations of JAVA_OPTS (-Xmx, -Xms, all types of GC) and nothing worked, Is the Java process not releasing memory back to OS? Is there any way to force it to do so?
1)
I suggest you to set -Xmx2100m and observe heap usage under load.
JVM may take as much OS memory as it decide to be performant, until it reaches Xmx limit. In modern JVMs default Xmx is calculated upon total memory available in OS, so it may be large value.
I think your app does not have memory leak, your JVM simply allocate a lot of memory, because it can.
Observe your JVM thru jvisualvm.
2)
Second suggestion - do you use any JNI code? Does your app call any native library (ie. dll under windows)?
I am suspecting a native memory leak in my java code. Are there any tools which do native memory profiling? Also, does any tool support native memory analysis of a running java process?
Thanks!!
Edit:
I have already tried Memory Validator and Purify but it seems they support only 32-bit processes. Is there some tool similar to the above ones which can simply attach to a running windows process and give us native memory analysis for that particular process?
The Troubleshooting guide for Java SE 6 with Hotspot VM contains a fairly elaborate section on techniques to aid in detecting native memory leaks. These include:
wrapping all memory allocation and deallocation calls to track the amount of memory used.
relying on platform specific support like the debug support provided by the Microsoft Visual C++ compiler or on mtrace (and MALLOC_TRACE) to debug memory allocations on Linux.
using memory leak analysis tools like Rational Purify.
among others. Notably, the article mentions that no ideal solution exists for all platforms.
Also, consider using the -Xcheck:jni flag that appears to be available in most JVMs. The -X flag itself indicates that the flag is non-standard, but the flag appears to be available in the IBM JDK, Oracle JRockit R28, and even the Oracle/Sun JVM. Enabling the flag switches on a mode where wrappers are added around JNI calls, thereby allowing you to track illegal arguments passed to JVM calls as noted in the JNI programmers' guide and specification. While, it's use in detecting memory leaks is subjective, it would definitely help if you suspect that the leak is being caused due to invalid parameters being issued.
AFAIK you can't do it with Java tools like JProfiler, JVisualVM etc. If you have memory leak in native code use tools for native code. You ie. can run it from C (i.e. loading jvm.dll). You can look at this articles finding memory leaks using Visual Studio or Memory Leak Detection in C++ (Linux)
Note: of course if you leak is connected to heap leak (forgot about deleteglobalref) you can find it with Java tools, but it is very rare in JNI.
I have been working on an open-source project named "MySafe" (https://github.com/serkan-ozal/mysafe) It basicly intercepts and monitors "Unsafe" calls. (In fact, it makes more than). With version 2.0, it can be useful for tracking and detecting "Unsafe" based native memory leaks.
Demo code: https://github.com/serkan-ozal/mysafe/blob/master/src/test/java/tr/com/serkanozal/mysafe/NativeMemoryLeakHuntingDemo.java
Diagram showing source of the leak: https://github.com/serkan-ozal/mysafe/blob/master/src/test/resources/native-memory-leak-hunting.png
To diagnose native memory leak, JIT code symbol mapping and Linux recent profiling tools are required: perf, perf-map-agent and bcc.
Please refer to details in related answer https://stackoverflow.com/a/52767721/737790
Many thanks to Brendan Gregg
These are tools you can use for debugging
libtcmalloc HPROF: For heap profiling
The jcmd Utility, PSS for the process: Can help in confirming a native leak.
Native Memory Tracking: Tracking native memory leak in JVM (only works for allocation inside JVM)
Core dump analysis, pmap and gdb checking for anon blocks and process memory overtime
-Xcheck:jni
Further details can be found out here
https://www.bro-code.in/blog/java-debugging-native-memory-leak
http://www.oracle.com/technetwork/java/javase/memleaks-137499.html#gbyvk
I'm a big fan of JProfiler. That's the best tool for profiling and memory leaks. It's fairly cheap relative to most tools, really easy to learn, and lots of features.
http://www.ej-technologies.com/products/jprofiler/overview.html
How do we track memory allocation in a Java application that makes use of JNI layer. More specifically if the underlying C code does some memory allocation, can we track those allocations using some profiler [Eclipse/Netbeans].
It should be possible to track those allocations using a native framework like Valgrind.
Running a JNI application in the Sun VM under Valgrind
I have a standalone program that I run locally, it is meant to be a server type program running 24/7. Recently I found that it has a memory leak, right now our only solution is to restart it every 4 hours. What is the best way to go about finding this memory leak? Which tool and method should we use?
If you are using Java from Sun and you use at least Java 6 update 10 (i.e. the newest), then try running jvisualvm from the JDK on the same machine as your program is running, and attach to it and enable profiling.
This is most likely the simplest way to get started.
When it comes to hunting memory problems, I use SAP Memory Analyzer Eclipse Memory Analyser (MAT), a Heap Dump analysis tool.
The Memory Analyzer provides a general purpose toolkit to analyze Java heap dumps. Besides heap walking and fast calculation of retained sizes, the Eclipse tool reports leak suspects and memory consumption anti-patterns. The main area of application are Out Of Memory Errors and high memory consumption.
Initiated by SAP, the project has since been open sourced and is now know as Eclipse Memory Analyser. Check out the Getting Started page and especially the Finding Memory Leaks section (I'm pasting it below because I fixed some links):
Start by running the leak report to automatically check for memory leaks.
This blog details How to Find a Leaking Workbench Window.
The Memory Analyzer grew up at SAP. Back then, Krum blogged about Finding Memory Leaks with SAP Memory Analyzer. The content is still relevant!
This is probably the best tool you can get (even for money) for heap dump analysis (and memory leaks).
PS: I do not work for SAP/IBM/Eclipse, I'm just a very happy MAT user with positive feedback.
You need a memory profiler. I recommend trying the Netbeans profiler.
One approach would be to take heap dumps on a regular basis, then trend the instance counts of your classes to try to work out which objects are being consistently created but not collected.
Another would be to switch off parts of your app to try to narrow down where the problem is.
Look at tools like jmap and jhat.
You might look up JMX and the jconsole app that ships with Java. You can get some interesting statistics out-of-the-box, and adding some simple instrumentation to your classes can provide a whole lot more.
As already stated jvisualvm is a great way to get started, but once you know what is leaking you may need to find what is holding references to the objects in question for which I'd recommend jmap and jhat, e.g
jmap -dump:live,file=heap.dump.out,format=b <pid>
and
jhat heap.dump.out
where <pid> is easily found from jvisualvm. Then in a browser navigate to localhost:7000 and begin exploring.
You need to try and capture Java heap dump which is a memory print of the Java process.
It's a critical process for memory consumption optimisation and finding memory leaks.
Java heap dump is an essential object for diagnosing memory-linked issues including java.lang.OutOfMemoryError, Garbage Collection issues, and memory leaks which are all part of Java web development process
For clarity, a Heap dump contains information such as Java classes and objects in a heap during instant of taking the snapshot.
To do it, you need to run jmap -dump:file=myheap.bin <program pid>.
To learn more about how to capture Java heat dumps, check out: https://javatutorial.net/capture-java-heap-dump