So far I have learned about generating thread dump and heap dump using jstack and and jmap respectively.
However, jstack thread dump contains only texts describing the stack on each thread. And opening heap dump (.hprof file) with Java VisualVM only shows the objects allocated in the heap.
What I actually want is to be able see the stack, to switch to particular stack frame, and watch local variables. This kind of post-mortem debugging can be done normally with tools like WinDbg, gdb and a core file (for a native C++ program.)
I wonder if such 'core' file (which will allow me to debug in non-live environment) exists in Java?
Java does. If you are using an IBM VM, use com.ibm.jvm.Dump.SystemDump() to programatically generate a dump. This can be debugged using a debugger. I believe "kill"ing your Java process should generate a system dump too. For Unix use kill -4 pid where pid is the process id and could be queried by typing in top | grep java if you have 1 VM instance running.
You could also add -Xdump:system or -Xdump:heap etc to your java command line to filter events and generate dumps on certain events like VM Stop (-Xdump:system:events=vmstop), full garbage collections(-Xdump:system:events=fullgc), etc. Note, depending on your heap size, generating a dump on a full GC is may not be a good idea (i.e you might create 50 dumps withing 20 seconds if you heap grows from 4M to around 60M in 20 seconds ) so you could add a counter like -Xdump:system:events=fullgc,range=50..55 which would generate 5 cores between the 50th to the 55th full garbage collect.
I've found relevant information in a Sun forum and in an SO discussion: I have not had much luck with it, but it might work in your case.
Note: some of the tools mentioned are Java tools, but are unsupported and are not available on Windows versions of the JDK.
I don't think such a dump mechanism exists in standard Java.
Some operating systems (for example Solaris mdb or gdb on Linux) support using the normal native debugger on dump files, with some special support for showing Java stack frames. But this is pretty hardcore and probably not what you want, since it is not well integrated with the Java Debugger.
Related
I have a Java web server running as a Windows service.
I use Tomcat 8 with Java 1.8.*
For a few months now, I've detected that the memory usage is increasing quite rapidly. I cannot make up for sure if it's heap or stack.
The process starts with ~200MB and after a week or so, it can reach up to 2GB.
Shortly after it will generate OutOfMemory exception (the memory usage will be 2GB - 2.5GB).
This has repeated multiple times on multiple environments.
I would like to know if there's a way to monitor the process and view it's internal memory usage, even to the level of viewing which objects are using the most amount of memory.
Can 'Java Native Memory Tracking' be used for this?
This will help me to detect any memory leaks that might cause this.
Thanks in advance.
To monitor the memory usage of a Java process, I'd use a JMX client such as JVisualVM, which is bundled with the Oracle JDK:
https://visualvm.java.net/jmx_connections.html
To identify the cause of a memory leak, I'd instruct the JVM to take a heap dump when it runs out of memory (on the Oracle JVM, this can be accomplished by specifying -XX:-HeapDumpOnOutOfMemoryError when starting your Java program), and then analyze that heap dump using a tool such as Eclipse MAT.
quoting:
the process starts with ~200MB and after a week or so, it can reach up to 2GB. Shortly after it will generate OutOfMemory exception (the memory usage will be 2GB - 2.5GB).
The problem might not be as simple as seeing what java objects you have got in JVisualVM (e.g millions of strings)
What you need to do is identify the code that leaks.
One way you could do that is to force the execution of particular code and then monitor the memory.
The easiest way to force the execution of code inside classes/objects is to use a tool like https://github.com/lorenzoongithub/nudge4j (particularly since you are on java 8)
alternatively you could just wire up nashorn to a command line or run your progam via jjs https://docs.oracle.com/javase/8/docs/technotes/guides/scripting/nashorn/shell.html
I have Java application, which, unfortunately, begins to consume quite big amounts of memory after some time. To complicate things, it's not only Java application, it is also JavaFX 2 application.
I suspect that there is some memory leak, maybe even in underlying JavaFX calls and native libs.
The ideal solution would be to get a dump of all java objects at some moment (with their memory usage), and then analyze that dump. Is there some way to achieve this?
Use jmap -heap:format=b <process-id> to create a binary dump of the heap which can then be loaded into several tools - my favorite being the "Eclipse Memory Analyzer"
There are lots of ways to get a heap dump, starting with simple tools like jmap to more fancy stuff like JVisualVM or even commerical tools as JProfiler. Correctly interpreting those dumps can be tricky though, so you might want to post exactly what you are looking for. Are going hunting for a memory leak, or are you interested in getting a general feel for your application?
You can use jvisualvm. It has plugin to see live memory and get a dump out of it.
I just re-discovered this article (archive.org archive) when researching ways to grab "JVM state right at this moment" - after a heap I pulled with jmap was about half the size of what the MBeans reported. I'll add it for completeness:
su $JVM_OWNER -c "gcore -o /tmp/jvm.core $YOUR_JVM_PID"
su $JVM_OWNER -c "jmap -dump:format=b,file=jvm.hprof /usr/bin/java /tmp/jvm.core"
Requires gdb installed (for gcore) and a JDK installation (for jmap). Also note you'd might need to adjust /usr/bin/java to the path of the JVM used for the process.
In a java heap dump how do I know exactly where in the code/which thread caused the dump?
For reading the memory dumps:
I would recommend you to try "eclipse memory analyzer" From here
Another option (free) would be opening this with JVisualVM (available at $JAVA_HOME/bin)
jhat is cool too but was already recommended :)
Now, you're asking about the thread that caused memory heap-dump and not about how to proceed with memory dump...
It depends on how did you obtain the memory dump.
There are different ways to obtain the dump.
On your process you can instruct the JVM to produce memory dump once OutOfMemory error is encountered, in this case I believe it will be the JVM itself.
You can trigger the heap dump creation from the MBean given you have a JMX Server running along with your JVM
Example
You can even use system calls (on linux) externally to your application: kill -3 _YOUR_JAVA_PROCESS_ID_ will generation the heapdump.
But I hardly can imagine why would you need such an information. Later in comments you mention 'exact line of code' but these ways are usually external to your JVM... Are you sure you need a line of code that generated heap dump itself, or you're trying to identify the real issue?
Hope this helps
In java you create object some where, use it in many places and the let GC to collect it back. There is no single line causing a leak.
What you should look for in tools like MAP is objects count and heap used by them. Pick each of the target class and see why they are not garbage collected. (some one is holding reference more than needed- say a static field )
You may find instructions from this page more useful - http://scn.sap.com/people/krum.tsvetkov/blog/2007/07/02/finding-memory-leaks-with-sap-memory-analyzer (also linked from MAT homepage)
Try to use Java Heap Analysis Tool(jhat) or jconsole (that is in your JAVA_HOME\bin).
I've got a java application that half the time just hangs, and the other half the JVM crashes. Is there a tool I can use to see what's going on that makes it hang and/or crash?
I'm using CentOS 5.6
For starters I would suggest JVisualVM. It comes with the JDK, so you should just need to type jvisualvm into the command line to start it.
Once it starts, you can connect to a running JVM, so you should be able to connect to your hung Java process and inspect the stack dump for all its running threads as well as the contents of the heap.
Other useful built-in tools include:
jps lists process ids of running java processes
jstack prints a stack dump for each thread in the specified JVM process
jmap generates a heap dump for the specified JVM process (jvisualvm can also generate heap dumps)
jhat analyzes heap dumps generated with jmap or jvisualvm
Of couse, there are also more sophisticated profilers available. JProfiler is quite highly regarded.
There are two different cases.
Application crash:
Was that an OOM? NPE? What was the exception? If there was jvm crash you will see hs_err_.log (http://java.sun.com/j2se/1.5/pdf/jdk50_ts_guide.pdf)
Looking at the file you may see if your own JNI caused a crash or JVM bug.
Application Hang: I would start with visualvm or jstat (both are part of JDK). You can see current state of threads and check if there is any application error..
Other linux tools that could help to see inside process:
lsof : you can check if the process opened too many files
strace: see current activity from system call point of view.
Oracle tools documentation provides pretty neat listing. It also links Operating System Specific tools
In these cases(hang, freeze, ...) you have to analyze an heap dump to try to figure out what's happening in your application , you can use JVisualVM to take the dump, or you can add the appropriate JVM parameter to dump the content of the heap in the case of a crash.
I have a HotSpot JVM heap dump that I would like to analyze. The VM ran with -Xmx31g, and the heap dump file is 48 GB large.
I won't even try jhat, as it requires about five times the heap memory (that would be 240 GB in my case) and is awfully slow.
Eclipse MAT crashes with an ArrayIndexOutOfBoundsException after analyzing the heap dump for several hours.
What other tools are available for that task? A suite of command line tools would be best, consisting of one program that transforms the heap dump into efficient data structures for analysis, combined with several other tools that work on the pre-structured data.
Normally, what I use is ParseHeapDump.sh included within Eclipse Memory Analyzer and described here, and I do that onto one our more beefed up servers (download and copy over the linux .zip distro, unzip there). The shell script needs less resources than parsing the heap from the GUI, plus you can run it on your beefy server with more resources (you can allocate more resources by adding something like -vmargs -Xmx40g -XX:-UseGCOverheadLimit to the end of the last line of the script.
For instance, the last line of that file might look like this after modification
./MemoryAnalyzer -consolelog -application org.eclipse.mat.api.parse "$#" -vmargs -Xmx40g -XX:-UseGCOverheadLimit
Run it like ./path/to/ParseHeapDump.sh ../today_heap_dump/jvm.hprof
After that succeeds, it creates a number of "index" files next to the .hprof file.
After creating the indices, I try to generate reports from that and scp those reports to my local machines and try to see if I can find the culprit just by that (not just the reports, not the indices). Here's a tutorial on creating the reports.
Example report:
./ParseHeapDump.sh ../today_heap_dump/jvm.hprof org.eclipse.mat.api:suspects
Other report options:
org.eclipse.mat.api:overview and org.eclipse.mat.api:top_components
If those reports are not enough and if I need some more digging (i.e. let's say via oql), I scp the indices as well as hprof file to my local machine, and then open the heap dump (with the indices in the same directory as the heap dump) with my Eclipse MAT GUI. From there, it does not need too much memory to run.
EDIT:
I just liked to add two notes :
As far as I know, only the generation of the indices is the memory intensive part of Eclipse MAT. After you have the indices, most of your processing from Eclipse MAT would not need that much memory.
Doing this on a shell script means I can do it on a headless server (and I normally do it on a headless server as well, because they're normally the most powerful ones). And if you have a server that can generate a heap dump of that size, chances are, you have another server out there that can process that much of a heap dump as well.
First step: increase the amount of RAM you are allocating to MAT. By default it's not very much and it can't open large files.
In case of using MAT on MAC (OSX) you'll have file MemoryAnalyzer.ini file in MemoryAnalyzer.app/Contents/MacOS. It wasn't working for me to make adjustments to that file and have them "take". You can instead create a modified startup command/shell script based on content of this file and run it from that directory. In my case I wanted 20 GB heap:
./MemoryAnalyzer -vmargs -Xmx20g --XX:-UseGCOverheadLimit ... other params desired
Just run this command/script from Contents/MacOS directory via terminal, to start the GUI with more RAM available.
I suggest trying YourKit. It usually needs a little less memory than the heap dump size (it indexes it and uses that information to retrieve what you want)
The accepted answer to this related question should provide a good start for you (if you have access to the running process, generates live jmap histograms instead of heap dumps, it's very fast):
Method for finding memory leak in large Java heap dumps
Most other heap analysers (I use IBM http://www.alphaworks.ibm.com/tech/heapanalyzer) require at least a percentage of RAM more than the heap if you're expecting a nice GUI tool.
Other than that, many developers use alternative approaches, like live stack analysis to get an idea of what's going on.
Although I must question why your heaps are so large? The effect on allocation and garbage collection must be massive. I'd bet a large percentage of what's in your heap should actually be stored in a database / a persistent cache etc etc.
This person http://blog.ragozin.info/2015/02/programatic-heapdump-analysis.html
wrote a custom "heap analyzer" that just exposes a "query style" interface through the heap dump file, instead of actually loading the file into memory.
https://github.com/aragozin/heaplib
Though I don't know if "query language" is better than the eclipse OQL mentioned in the accepted answer here.
The latest snapshot build of Eclipse Memory Analyzer has a facility to randomly discard a certain percentage of objects to reduce memory consumption and allow the remaining objects to be analyzed. See Bug 563960 and the nightly snapshot build to test this facility before it is included in the next release of MAT. Update: it is now included in released version 1.11.0.
A not so well known tool - http://dr-brenschede.de/bheapsampler/ works well for large heaps. It works by sampling so it doesn't have to read the entire thing, though a bit finicky.
This is not a command line solution, however I like the tools:
Copy the heap dump to a server large enough to host it. It is very well possible that the original server can be used.
Enter the server via ssh -X to run the graphical tool remotely and use jvisualvm from the Java binary directory to load the .hprof file of the heap dump.
The tool does not load the complete heap dump into memory at once, but loads parts when they are required. Of course, if you look around enough in the file the required memory will finally reach the size of the heap dump.
I came across an interesting tool called JXray. It provides limited evaluation trial license. Found it very useful to find memory leaks. You may give it a shot.
Try using jprofiler , its works good in analyzing large .hprof, I have tried with file sized around 22 GB.
https://www.ej-technologies.com/products/jprofiler/overview.html
$499/dev license but has a free 10 day evaluation
When the problem can be "easily" reproduced, one unmentioned alternative is to take heap dumps before memory grows that big (e.g., jmap -dump:format=b,file=heap.bin <pid>).
In many cases you will already get an idea of what's going on without waiting for an OOM.
In addition, MAT provides a feature to compare different snapshots, which can come handy (see https://stackoverflow.com/a/55926302/898154 for instructions and a description).