Is there a tool or way to detect which part of a Java web application/web server leaks native memory in RedHat Linux? Is there any free profiling tool that can be used to find out what cause memory consumption to increase within the application/server?
Related
I have a server java app use jetty websocket server and bittorrent-tracker for websocket client
The issue is when I run with many client it has a memory leak in my java app
I check the memory usage of java app by "htop" and "jcmd PID VM.native_memory"
The RES in "htop" show 6.4G
The memory committed in "jcmd" show 2G
The jetty websocket version I use is jetty-9.4.15.v20190215, I had try newer version jetty-9.4.44.v20210927 but still memory leak
I had tested with client build from jetty websocket-client and memory leak doesn't happen
I also test with following options
Only use lib jetty websocket on my server java app
When handle onMessage I just receive messages and not take any process or response
=> Basically the server only receives connect from client and memory leak still happen.
The question is
Where cause memory leak ?
Why physical memory(RES) in "htop" larger than committed memory from jcmd ?
"Where cause memory leak ?"
It is probably your code. For example, if websockets are not being closed correctly on the server side, they may hang around consuming various resources ... including off-heap buffers and so on.
It is not inconceivable that there is a bug in the Jetty code that causes this, but if there was, one would expect that other people will have reported it, etcetera. (Have you checked the Jetty issue tracker for confirmed reports of memory leaks in the Jetty websocket code?)
Either way you will need to do some investigation.
Why physical memory(RES) in "htop" larger than memory usage in JVM ?
That is not unusual. A typical JVM uses a lot of memory in addition to the Java heap. This usage includes:
The memory used by the JVM executable, and native libraries that it uses.
Metaspace ... which holds compiled Java code, and so on.
Off-heap allocations requested by native code for various purposes.
Memory mapped files and so on.
I have a Java web server running as a Windows service.
I use Tomcat 8 with Java 1.8.*
For a few months now, I've detected that the memory usage is increasing quite rapidly. I cannot make up for sure if it's heap or stack.
The process starts with ~200MB and after a week or so, it can reach up to 2GB.
Shortly after it will generate OutOfMemory exception (the memory usage will be 2GB - 2.5GB).
This has repeated multiple times on multiple environments.
I would like to know if there's a way to monitor the process and view it's internal memory usage, even to the level of viewing which objects are using the most amount of memory.
Can 'Java Native Memory Tracking' be used for this?
This will help me to detect any memory leaks that might cause this.
Thanks in advance.
To monitor the memory usage of a Java process, I'd use a JMX client such as JVisualVM, which is bundled with the Oracle JDK:
https://visualvm.java.net/jmx_connections.html
To identify the cause of a memory leak, I'd instruct the JVM to take a heap dump when it runs out of memory (on the Oracle JVM, this can be accomplished by specifying -XX:-HeapDumpOnOutOfMemoryError when starting your Java program), and then analyze that heap dump using a tool such as Eclipse MAT.
quoting:
the process starts with ~200MB and after a week or so, it can reach up to 2GB. Shortly after it will generate OutOfMemory exception (the memory usage will be 2GB - 2.5GB).
The problem might not be as simple as seeing what java objects you have got in JVisualVM (e.g millions of strings)
What you need to do is identify the code that leaks.
One way you could do that is to force the execution of particular code and then monitor the memory.
The easiest way to force the execution of code inside classes/objects is to use a tool like https://github.com/lorenzoongithub/nudge4j (particularly since you are on java 8)
alternatively you could just wire up nashorn to a command line or run your progam via jjs https://docs.oracle.com/javase/8/docs/technotes/guides/scripting/nashorn/shell.html
I think I may have a memory leak in a servlet application running in production on jetty 8.1.7.
Is there a way of seeing how much heap memory is actually being used at an instance of time, not the max memory allocated with -Xmx, but the actual amount of memory being used.
Can I force a garbage collection to occur for an application running within jetty
yes, both are easily achievable using: VisualVM (see: http://docs.oracle.com/javase/6/docs/technotes/guides/visualvm/monitor_tab.html) This one is shipped with Oracle JDK by default (=> no extra installation required)
However for the memory leak detection, I'd suggest to do memory dump and analyze it later with eclipse MAT ( http://www.eclipse.org/mat/ ) as it has quite nice UI visualizing java memory dumps.
EDIT:
For the ssh only access, yes you can use the mentioned two tools. However you need to run them on the machine with running window manager and remotely connect over ssh to the other machine (you need to have java on both of these machines):
For visualVM: you need to have VisualVM running on one maching and via the ssh connect to remote one, see: VisualVM over ssh
and for the memory dump: use jmap (for sample usage see: http://kadirsert.blogspot.de/2012/01/…) afterwards download the dump file and load if locally to eclipse MAT
enable jmx and connect up to it using jconsole
http://wiki.eclipse.org/Jetty/Tutorial/JMX
You can call System.gc(). That will typically perform a full GC ... but this facility can be disabled. (There is a JVM option to do this with HotSpot JVMs.)
However, if your problem is a memory leak, running the GC won't help. In fact, it is likely to make your server even slower than it currently is.
You can also monitor the memory usage (in a variety of ways - see other Answers) but that only gives you evidence that a memory leak might leak.
What you really need to do is find and fix the cause of the memory leak.
Reference:
How to find a Java Memory Leak
You can use jvisualvm.exe which is under the %JAVA_HOME%\bin folder. By using this application you can monitor memory usage and can force gc.
Can you suggest me a tool for monitoring multiple java processes
running on a Windows 7 machine.
More specifically I have like 10 similar applications running with default JVM settings.
I need information for the memory allocation over time, to identify possible memory leaks and to adjust more resources somewhere if needed.
Some free tools with logs and graphs will be perfect.
How about the standard JVM JVisualVM or VisualVM
Problem :
I have setted Xms512m and Xmx1024m for running application which intern use C++ native layer for performing other operation, I am getting OutOfMemory exception when running application. I need to know C++ uses which memory (ie from assigned memory Xms512m and Xmx1024m or it uses other than this setted memory).
How to get heap space and stack space for Java and C++ code while running application separately.
The memory allocated by native code is not in the Java Heap. Your OutOfMemory Exception caused by the java application. OOM is a clue to get you a sign that java app may use more than 1024mb memory. You can make it larger or consider the memory leak problem.
There are some links about memory leak detection:
Eclipse Memory Analyser (MAT) - Tutorial
10 points about Java Heap Space or Java Heap Memory
You can do it simply:
get java app pid by jps
qty:~ qrtt1$ jps
4437 start.jar
10470 Jps
get heap dump data by jmap
jmap -dump:format=b,file=my_app_heap_data.hprof 4437
use mat to open it, like this:
You may use jconsole or VisualVM to see heap usage of the application.
You can Use below monitoring applications:
jconsole
javamelody
visualvm
The memory allocated to your JVM and the memory used by JNI and native applications are completely different. Tools like VisualVM and Jprofiler can help you determine JVM specific heap usage. If however the memory leak is from JNI or a native application, you should use options like -Xrunjnichk (available on the IBM JDK) to debug JNI calls.