How do we track memory allocation in a Java application that makes use of JNI layer. More specifically if the underlying C code does some memory allocation, can we track those allocations using some profiler [Eclipse/Netbeans].
It should be possible to track those allocations using a native framework like Valgrind.
Running a JNI application in the Sun VM under Valgrind
Related
JVM native memory leak is suspected, so I want to track memory.
I tried Native tracking using the JVM tuning flag as follows, but I couldn't find a memory leak point.
java -XX:NativeMemoryTracking=detail
Probably because of the following reasons in the document.
NMT in this release does not track third party native code memory allocations and JDK class libraries.
Also, this release does not include NMT MBean in HotSpot for JMC.
Is there any other way to track JVM native memory?
There are several tools that you can use.
I'll refer to this excellent answer by apangin: Java using much more memory than heap size (or size correctly Docker memory limit)
I strongly encourage you to read all of that but for your question, this in particular is relevant:
There are tools and techniques for investigating issues with Java memory consumption: Native Memory Tracking, pmap, jemalloc, async-profiler.
I have a Java web server running as a Windows service.
I use Tomcat 8 with Java 1.8.*
For a few months now, I've detected that the memory usage is increasing quite rapidly. I cannot make up for sure if it's heap or stack.
The process starts with ~200MB and after a week or so, it can reach up to 2GB.
Shortly after it will generate OutOfMemory exception (the memory usage will be 2GB - 2.5GB).
This has repeated multiple times on multiple environments.
I would like to know if there's a way to monitor the process and view it's internal memory usage, even to the level of viewing which objects are using the most amount of memory.
Can 'Java Native Memory Tracking' be used for this?
This will help me to detect any memory leaks that might cause this.
Thanks in advance.
To monitor the memory usage of a Java process, I'd use a JMX client such as JVisualVM, which is bundled with the Oracle JDK:
https://visualvm.java.net/jmx_connections.html
To identify the cause of a memory leak, I'd instruct the JVM to take a heap dump when it runs out of memory (on the Oracle JVM, this can be accomplished by specifying -XX:-HeapDumpOnOutOfMemoryError when starting your Java program), and then analyze that heap dump using a tool such as Eclipse MAT.
quoting:
the process starts with ~200MB and after a week or so, it can reach up to 2GB. Shortly after it will generate OutOfMemory exception (the memory usage will be 2GB - 2.5GB).
The problem might not be as simple as seeing what java objects you have got in JVisualVM (e.g millions of strings)
What you need to do is identify the code that leaks.
One way you could do that is to force the execution of particular code and then monitor the memory.
The easiest way to force the execution of code inside classes/objects is to use a tool like https://github.com/lorenzoongithub/nudge4j (particularly since you are on java 8)
alternatively you could just wire up nashorn to a command line or run your progam via jjs https://docs.oracle.com/javase/8/docs/technotes/guides/scripting/nashorn/shell.html
Is there a tool or way to detect which part of a Java web application/web server leaks native memory in RedHat Linux? Is there any free profiling tool that can be used to find out what cause memory consumption to increase within the application/server?
Problem :
I have setted Xms512m and Xmx1024m for running application which intern use C++ native layer for performing other operation, I am getting OutOfMemory exception when running application. I need to know C++ uses which memory (ie from assigned memory Xms512m and Xmx1024m or it uses other than this setted memory).
How to get heap space and stack space for Java and C++ code while running application separately.
The memory allocated by native code is not in the Java Heap. Your OutOfMemory Exception caused by the java application. OOM is a clue to get you a sign that java app may use more than 1024mb memory. You can make it larger or consider the memory leak problem.
There are some links about memory leak detection:
Eclipse Memory Analyser (MAT) - Tutorial
10 points about Java Heap Space or Java Heap Memory
You can do it simply:
get java app pid by jps
qty:~ qrtt1$ jps
4437 start.jar
10470 Jps
get heap dump data by jmap
jmap -dump:format=b,file=my_app_heap_data.hprof 4437
use mat to open it, like this:
You may use jconsole or VisualVM to see heap usage of the application.
You can Use below monitoring applications:
jconsole
javamelody
visualvm
The memory allocated to your JVM and the memory used by JNI and native applications are completely different. Tools like VisualVM and Jprofiler can help you determine JVM specific heap usage. If however the memory leak is from JNI or a native application, you should use options like -Xrunjnichk (available on the IBM JDK) to debug JNI calls.
I am suspecting a native memory leak in my java code. Are there any tools which do native memory profiling? Also, does any tool support native memory analysis of a running java process?
Thanks!!
Edit:
I have already tried Memory Validator and Purify but it seems they support only 32-bit processes. Is there some tool similar to the above ones which can simply attach to a running windows process and give us native memory analysis for that particular process?
The Troubleshooting guide for Java SE 6 with Hotspot VM contains a fairly elaborate section on techniques to aid in detecting native memory leaks. These include:
wrapping all memory allocation and deallocation calls to track the amount of memory used.
relying on platform specific support like the debug support provided by the Microsoft Visual C++ compiler or on mtrace (and MALLOC_TRACE) to debug memory allocations on Linux.
using memory leak analysis tools like Rational Purify.
among others. Notably, the article mentions that no ideal solution exists for all platforms.
Also, consider using the -Xcheck:jni flag that appears to be available in most JVMs. The -X flag itself indicates that the flag is non-standard, but the flag appears to be available in the IBM JDK, Oracle JRockit R28, and even the Oracle/Sun JVM. Enabling the flag switches on a mode where wrappers are added around JNI calls, thereby allowing you to track illegal arguments passed to JVM calls as noted in the JNI programmers' guide and specification. While, it's use in detecting memory leaks is subjective, it would definitely help if you suspect that the leak is being caused due to invalid parameters being issued.
AFAIK you can't do it with Java tools like JProfiler, JVisualVM etc. If you have memory leak in native code use tools for native code. You ie. can run it from C (i.e. loading jvm.dll). You can look at this articles finding memory leaks using Visual Studio or Memory Leak Detection in C++ (Linux)
Note: of course if you leak is connected to heap leak (forgot about deleteglobalref) you can find it with Java tools, but it is very rare in JNI.
I have been working on an open-source project named "MySafe" (https://github.com/serkan-ozal/mysafe) It basicly intercepts and monitors "Unsafe" calls. (In fact, it makes more than). With version 2.0, it can be useful for tracking and detecting "Unsafe" based native memory leaks.
Demo code: https://github.com/serkan-ozal/mysafe/blob/master/src/test/java/tr/com/serkanozal/mysafe/NativeMemoryLeakHuntingDemo.java
Diagram showing source of the leak: https://github.com/serkan-ozal/mysafe/blob/master/src/test/resources/native-memory-leak-hunting.png
To diagnose native memory leak, JIT code symbol mapping and Linux recent profiling tools are required: perf, perf-map-agent and bcc.
Please refer to details in related answer https://stackoverflow.com/a/52767721/737790
Many thanks to Brendan Gregg
These are tools you can use for debugging
libtcmalloc HPROF: For heap profiling
The jcmd Utility, PSS for the process: Can help in confirming a native leak.
Native Memory Tracking: Tracking native memory leak in JVM (only works for allocation inside JVM)
Core dump analysis, pmap and gdb checking for anon blocks and process memory overtime
-Xcheck:jni
Further details can be found out here
https://www.bro-code.in/blog/java-debugging-native-memory-leak
http://www.oracle.com/technetwork/java/javase/memleaks-137499.html#gbyvk
I'm a big fan of JProfiler. That's the best tool for profiling and memory leaks. It's fairly cheap relative to most tools, really easy to learn, and lots of features.
http://www.ej-technologies.com/products/jprofiler/overview.html