Develop Reference

Tool for analyzing large Java heap dumps

I have a HotSpot JVM heap dump that I would like to analyze. The VM ran with -Xmx31g, and the heap dump file is 48 GB large.
I won't even try jhat, as it requires about five times the heap memory (that would be 240 GB in my case) and is awfully slow.
Eclipse MAT crashes with an ArrayIndexOutOfBoundsException after analyzing the heap dump for several hours.
What other tools are available for that task? A suite of command line tools would be best, consisting of one program that transforms the heap dump into efficient data structures for analysis, combined with several other tools that work on the pre-structured data.

Normally, what I use is ParseHeapDump.sh included within Eclipse Memory Analyzer and described here, and I do that onto one our more beefed up servers (download and copy over the linux .zip distro, unzip there). The shell script needs less resources than parsing the heap from the GUI, plus you can run it on your beefy server with more resources (you can allocate more resources by adding something like -vmargs -Xmx40g -XX:-UseGCOverheadLimit to the end of the last line of the script.
For instance, the last line of that file might look like this after modification
./MemoryAnalyzer -consolelog -application org.eclipse.mat.api.parse "$#" -vmargs -Xmx40g -XX:-UseGCOverheadLimit
Run it like ./path/to/ParseHeapDump.sh ../today_heap_dump/jvm.hprof
After that succeeds, it creates a number of "index" files next to the .hprof file.
After creating the indices, I try to generate reports from that and scp those reports to my local machines and try to see if I can find the culprit just by that (not just the reports, not the indices). Here's a tutorial on creating the reports.
Example report:
./ParseHeapDump.sh ../today_heap_dump/jvm.hprof org.eclipse.mat.api:suspects
Other report options:
org.eclipse.mat.api:overview and org.eclipse.mat.api:top_components
If those reports are not enough and if I need some more digging (i.e. let's say via oql), I scp the indices as well as hprof file to my local machine, and then open the heap dump (with the indices in the same directory as the heap dump) with my Eclipse MAT GUI. From there, it does not need too much memory to run.
EDIT:
I just liked to add two notes :
As far as I know, only the generation of the indices is the memory intensive part of Eclipse MAT. After you have the indices, most of your processing from Eclipse MAT would not need that much memory.
Doing this on a shell script means I can do it on a headless server (and I normally do it on a headless server as well, because they're normally the most powerful ones). And if you have a server that can generate a heap dump of that size, chances are, you have another server out there that can process that much of a heap dump as well.

First step: increase the amount of RAM you are allocating to MAT. By default it's not very much and it can't open large files.
In case of using MAT on MAC (OSX) you'll have file MemoryAnalyzer.ini file in MemoryAnalyzer.app/Contents/MacOS. It wasn't working for me to make adjustments to that file and have them "take". You can instead create a modified startup command/shell script based on content of this file and run it from that directory. In my case I wanted 20 GB heap:
./MemoryAnalyzer -vmargs -Xmx20g --XX:-UseGCOverheadLimit ... other params desired
Just run this command/script from Contents/MacOS directory via terminal, to start the GUI with more RAM available.

I suggest trying YourKit. It usually needs a little less memory than the heap dump size (it indexes it and uses that information to retrieve what you want)

The accepted answer to this related question should provide a good start for you (if you have access to the running process, generates live jmap histograms instead of heap dumps, it's very fast):
Method for finding memory leak in large Java heap dumps
Most other heap analysers (I use IBM http://www.alphaworks.ibm.com/tech/heapanalyzer) require at least a percentage of RAM more than the heap if you're expecting a nice GUI tool.
Other than that, many developers use alternative approaches, like live stack analysis to get an idea of what's going on.
Although I must question why your heaps are so large? The effect on allocation and garbage collection must be massive. I'd bet a large percentage of what's in your heap should actually be stored in a database / a persistent cache etc etc.

This person http://blog.ragozin.info/2015/02/programatic-heapdump-analysis.html
wrote a custom "heap analyzer" that just exposes a "query style" interface through the heap dump file, instead of actually loading the file into memory.
https://github.com/aragozin/heaplib
Though I don't know if "query language" is better than the eclipse OQL mentioned in the accepted answer here.

The latest snapshot build of Eclipse Memory Analyzer has a facility to randomly discard a certain percentage of objects to reduce memory consumption and allow the remaining objects to be analyzed. See Bug 563960 and the nightly snapshot build to test this facility before it is included in the next release of MAT. Update: it is now included in released version 1.11.0.

A not so well known tool - http://dr-brenschede.de/bheapsampler/ works well for large heaps. It works by sampling so it doesn't have to read the entire thing, though a bit finicky.

This is not a command line solution, however I like the tools:
Copy the heap dump to a server large enough to host it. It is very well possible that the original server can be used.
Enter the server via ssh -X to run the graphical tool remotely and use jvisualvm from the Java binary directory to load the .hprof file of the heap dump.
The tool does not load the complete heap dump into memory at once, but loads parts when they are required. Of course, if you look around enough in the file the required memory will finally reach the size of the heap dump.

I came across an interesting tool called JXray. It provides limited evaluation trial license. Found it very useful to find memory leaks. You may give it a shot.

Try using jprofiler , its works good in analyzing large .hprof, I have tried with file sized around 22 GB.
https://www.ej-technologies.com/products/jprofiler/overview.html
$499/dev license but has a free 10 day evaluation

When the problem can be "easily" reproduced, one unmentioned alternative is to take heap dumps before memory grows that big (e.g., jmap -dump:format=b,file=heap.bin <pid>).
In many cases you will already get an idea of what's going on without waiting for an OOM.
In addition, MAT provides a feature to compare different snapshots, which can come handy (see https://stackoverflow.com/a/55926302/898154 for instructions and a description).

How to analyze a large heapdump?

Is there a tool to analyze a large Java Heap dump (2GB), if one only can assign 1,5GB to the JVM? I can't believe the dump must be fully loaded into memory to be analyzed...
Eclipse MemoryAnalyzer fails, and the IBM tool also.
Do I need to use command line tools here now?

If it's a dev server, restrict the max heap size to something a 32-bit OS can handle. If it's in production, demand a 64-bit OS! If you can't get that, you can run jhat on the server (it has a web interface you can access on your own PC).

One solution is to install the MAT tool on the remote server and generate an HTML output of the analysis to download and view locally. This saves the headache of attempting to get X Windows installed on the remote machine and get all of the ssh tunneling sorted out (which is of course an option as well).
First, download and install the stand-alone Eclipse RCP Application. Then transfer to your server and unpack. Then determine how large the heap dump is and, if necessary, modify the MemoryAnalyzer.ini file to instantiate a JVM with enough RAM for your heap dump.
In this example, I have an 11GB heap dump and have modified the last two lines (adding -Xms)
-startup
plugins/org.eclipse.equinox.launcher_1.3.100.v20150511-1540.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.300.v20150602-1417
-vmargs
-Xmx16g
-Xms16g
Do an initial run to parse the heap dump. This will generate intermediary data that can be used by subsequent runs to make future analysis faster.
./ParseHeapDump.sh /path/to/heap-dump
After that completes, you can run any of a number of different analysis on the data. The following is an illustration of how to search for memory leak suspects.
./ParseHeapDump.sh /path/to/heap-dump org.eclipse.mat.api:suspec

Unfortunately Eclipse MAT and all heap dump analysis tools loads entire heap dump into memory in order to do the analysis. If Eclipse MAT fails for you, you may try HeapHero tool. JHAT take lore more memory and time than Eclipse MAT to analyze heap dumps.

Get visual graph of heap memory usage in Java over a span of time

I am currently using Visual VM to monitor the heap memory usage of my Java application. However I would like to somehow see the heap memory usage over a span of time like for example a day and not just get a snapshot.I would like to be able to leave Visual VM or a tool on and let it log the memory usage and then later after one day, I can go back and see a graph of it. Is there a way to do this using Visual VM? If yes, how? If not, what tool can I used to do this?

Run your Java program with the following Java options:
-Xloggc:log.out -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
and download HPjmeter to visualize log.out.
Also see SUN's GC portal webpage for more options to run with. Since the data is written to a file, you won't have any problems collecting days or weeks worth of data. Of course, if you wish to visualize data with lots of information, you'll need to run HPjmeter with more memory.
Your other option is to use JConsole.

Try the Memory tab in JConsole. JConsole is also included with the Oracle JDK, like JVisualVM, so you should already have it. It has a time range of "all" which should work for what you want to do. It will look like this:

Core dump equivalent for Java

So far I have learned about generating thread dump and heap dump using jstack and and jmap respectively.
However, jstack thread dump contains only texts describing the stack on each thread. And opening heap dump (.hprof file) with Java VisualVM only shows the objects allocated in the heap.
What I actually want is to be able see the stack, to switch to particular stack frame, and watch local variables. This kind of post-mortem debugging can be done normally with tools like WinDbg, gdb and a core file (for a native C++ program.)
I wonder if such 'core' file (which will allow me to debug in non-live environment) exists in Java?

Java does. If you are using an IBM VM, use com.ibm.jvm.Dump.SystemDump() to programatically generate a dump. This can be debugged using a debugger. I believe "kill"ing your Java process should generate a system dump too. For Unix use kill -4 pid where pid is the process id and could be queried by typing in top | grep java if you have 1 VM instance running.
You could also add -Xdump:system or -Xdump:heap etc to your java command line to filter events and generate dumps on certain events like VM Stop (-Xdump:system:events=vmstop), full garbage collections(-Xdump:system:events=fullgc), etc. Note, depending on your heap size, generating a dump on a full GC is may not be a good idea (i.e you might create 50 dumps withing 20 seconds if you heap grows from 4M to around 60M in 20 seconds ) so you could add a counter like -Xdump:system:events=fullgc,range=50..55 which would generate 5 cores between the 50th to the 55th full garbage collect.

I've found relevant information in a Sun forum and in an SO discussion: I have not had much luck with it, but it might work in your case.
Note: some of the tools mentioned are Java tools, but are unsupported and are not available on Windows versions of the JDK.

I don't think such a dump mechanism exists in standard Java.

Some operating systems (for example Solaris mdb or gdb on Linux) support using the normal native debugger on dump files, with some special support for showing Java stack frames. But this is pretty hardcore and probably not what you want, since it is not well integrated with the Java Debugger.

General strategy to resolve Java memory leak?

I have a standalone program that I run locally, it is meant to be a server type program running 24/7. Recently I found that it has a memory leak, right now our only solution is to restart it every 4 hours. What is the best way to go about finding this memory leak? Which tool and method should we use?

If you are using Java from Sun and you use at least Java 6 update 10 (i.e. the newest), then try running jvisualvm from the JDK on the same machine as your program is running, and attach to it and enable profiling.
This is most likely the simplest way to get started.

When it comes to hunting memory problems, I use SAP Memory Analyzer Eclipse Memory Analyser (MAT), a Heap Dump analysis tool.
The Memory Analyzer provides a general purpose toolkit to analyze Java heap dumps. Besides heap walking and fast calculation of retained sizes, the Eclipse tool reports leak suspects and memory consumption anti-patterns. The main area of application are Out Of Memory Errors and high memory consumption.
Initiated by SAP, the project has since been open sourced and is now know as Eclipse Memory Analyser. Check out the Getting Started page and especially the Finding Memory Leaks section (I'm pasting it below because I fixed some links):
Start by running the leak report to automatically check for memory leaks.
This blog details How to Find a Leaking Workbench Window.
The Memory Analyzer grew up at SAP. Back then, Krum blogged about Finding Memory Leaks with SAP Memory Analyzer. The content is still relevant!
This is probably the best tool you can get (even for money) for heap dump analysis (and memory leaks).
PS: I do not work for SAP/IBM/Eclipse, I'm just a very happy MAT user with positive feedback.

You need a memory profiler. I recommend trying the Netbeans profiler.

One approach would be to take heap dumps on a regular basis, then trend the instance counts of your classes to try to work out which objects are being consistently created but not collected.
Another would be to switch off parts of your app to try to narrow down where the problem is.
Look at tools like jmap and jhat.

You might look up JMX and the jconsole app that ships with Java. You can get some interesting statistics out-of-the-box, and adding some simple instrumentation to your classes can provide a whole lot more.

As already stated jvisualvm is a great way to get started, but once you know what is leaking you may need to find what is holding references to the objects in question for which I'd recommend jmap and jhat, e.g
jmap -dump:live,file=heap.dump.out,format=b <pid>
and
jhat heap.dump.out
where <pid> is easily found from jvisualvm. Then in a browser navigate to localhost:7000 and begin exploring.

You need to try and capture Java heap dump which is a memory print of the Java process.
It's a critical process for memory consumption optimisation and finding memory leaks.
Java heap dump is an essential object for diagnosing memory-linked issues including java.lang.OutOfMemoryError, Garbage Collection issues, and memory leaks which are all part of Java web development process
For clarity, a Heap dump contains information such as Java classes and objects in a heap during instant of taking the snapshot.
To do it, you need to run jmap -dump:file=myheap.bin <program pid>.
To learn more about how to capture Java heat dumps, check out: https://javatutorial.net/capture-java-heap-dump