Develop Reference

Tool for debugging hangs in java application

I've got a java application that half the time just hangs, and the other half the JVM crashes. Is there a tool I can use to see what's going on that makes it hang and/or crash?
I'm using CentOS 5.6

For starters I would suggest JVisualVM. It comes with the JDK, so you should just need to type jvisualvm into the command line to start it.
Once it starts, you can connect to a running JVM, so you should be able to connect to your hung Java process and inspect the stack dump for all its running threads as well as the contents of the heap.
Other useful built-in tools include:
jps lists process ids of running java processes
jstack prints a stack dump for each thread in the specified JVM process
jmap generates a heap dump for the specified JVM process (jvisualvm can also generate heap dumps)
jhat analyzes heap dumps generated with jmap or jvisualvm
Of couse, there are also more sophisticated profilers available. JProfiler is quite highly regarded.

There are two different cases.
Application crash:
Was that an OOM? NPE? What was the exception? If there was jvm crash you will see hs_err_.log (http://java.sun.com/j2se/1.5/pdf/jdk50_ts_guide.pdf)
Looking at the file you may see if your own JNI caused a crash or JVM bug.
Application Hang: I would start with visualvm or jstat (both are part of JDK). You can see current state of threads and check if there is any application error..
Other linux tools that could help to see inside process:
lsof : you can check if the process opened too many files
strace: see current activity from system call point of view.
Oracle tools documentation provides pretty neat listing. It also links Operating System Specific tools

In these cases(hang, freeze, ...) you have to analyze an heap dump to try to figure out what's happening in your application , you can use JVisualVM to take the dump, or you can add the appropriate JVM parameter to dump the content of the heap in the case of a crash.

Collecting Java heapdump under load

I am running load against Tomcat 6 running on Java 6. I want to collect a heapdump of the Java heap while the Tomcat server is under load. I normally use jmap -dump to collect my heapdumps.
However, when I try to do this when Tomcat is handling a high load I find that the heapdump collection fails.
Is jmap the best tool for collecting a heap dump from a process under load? What are the possible causes which would cause jmap to fail to collect a heapdump?
If jmap is not the best tool - what is better?
It is entirely acceptable to me for jmap (or some other tool) to stop the world within the Java process while the heap dump is taken.

Is jmap the best tool for collecting a heap dump from a process under load?
I think: No it isn't. From this link:
NOTE - This utility is unsupported and
may or may not be available in future
versions of the JDK.

I've also found jmap can pretty temperamental. If you're having problems:
Try it again. It often manages to get a heap dump after a couple of attempts if it first fails
Use the -F option
Add -XX:+HeapDumpOnOutOfMemoryError as a standard configuration to proactively take heap dumps when an OOM error is thrown
Run Tomcat interactively and add the heap dump on ctrl-break option. This gives you a thread dump too, something you'll probably need anyway
If your heap size is especially large and you have a repeatable condition, temporarily lower your heap size. It makes the resulting file much easier to handle, takes less time and is more likely to succeed

I have found that running Tomcat with a JMX port allows me to take a remote heapdump using visualvm. This succeeded for me when jmap failed.

How do I analyze a .mdmp file?

I have java application that is crashing while in production. It doesn't do so in dev/QA. The jvm is creating a .mdmp file and a text file. How do I analyze the binary dump file? I googled but had no luck. We are using bea jrockit jvm 1.5 R27.

The .mdmp file is a Windows MiniDump file that you can only read with a debugger (like WinDbg). Typically you need the sources of the crashed application to really get some information out of the dump. So in your case you can't do much but contacting JRockit support.
Here a link to the Orace JRockit information about JVM crahes.

.mdmp files are the Windows equivalent of unix/linux core dumps. You can analyse them with WinDBG but if it's a Java process that has crashed most likely you'll want to use Java's own tools to analyse the crashed process.
If you want to look at the heap of the crashed Java process you can use a tool that ships with the JDK called jmap to extract a HPROF file from a .core or .mdmp and then load this into a memory analyser. Note also that some memory analyzers can load core dumps and Windows minidumps directly.
Related issue and the jmap docs
If you want to see the state of the threads then you can use a tool called jstack to print stack traces for every thread at the point the dump was created. jstack docs.

Core dump equivalent for Java

So far I have learned about generating thread dump and heap dump using jstack and and jmap respectively.
However, jstack thread dump contains only texts describing the stack on each thread. And opening heap dump (.hprof file) with Java VisualVM only shows the objects allocated in the heap.
What I actually want is to be able see the stack, to switch to particular stack frame, and watch local variables. This kind of post-mortem debugging can be done normally with tools like WinDbg, gdb and a core file (for a native C++ program.)
I wonder if such 'core' file (which will allow me to debug in non-live environment) exists in Java?

Java does. If you are using an IBM VM, use com.ibm.jvm.Dump.SystemDump() to programatically generate a dump. This can be debugged using a debugger. I believe "kill"ing your Java process should generate a system dump too. For Unix use kill -4 pid where pid is the process id and could be queried by typing in top | grep java if you have 1 VM instance running.
You could also add -Xdump:system or -Xdump:heap etc to your java command line to filter events and generate dumps on certain events like VM Stop (-Xdump:system:events=vmstop), full garbage collections(-Xdump:system:events=fullgc), etc. Note, depending on your heap size, generating a dump on a full GC is may not be a good idea (i.e you might create 50 dumps withing 20 seconds if you heap grows from 4M to around 60M in 20 seconds ) so you could add a counter like -Xdump:system:events=fullgc,range=50..55 which would generate 5 cores between the 50th to the 55th full garbage collect.

I've found relevant information in a Sun forum and in an SO discussion: I have not had much luck with it, but it might work in your case.
Note: some of the tools mentioned are Java tools, but are unsupported and are not available on Windows versions of the JDK.

I don't think such a dump mechanism exists in standard Java.

Some operating systems (for example Solaris mdb or gdb on Linux) support using the normal native debugger on dump files, with some special support for showing Java stack frames. But this is pretty hardcore and probably not what you want, since it is not well integrated with the Java Debugger.

Can I generate an HPROF file at will?

I have a java process which is acting dubiously. I'd like to see what's up using the various HPROF analysis tools.
How do I generate one on the fly?

Yes. You can generate an hprof file (containing heap memory usage) on the fly using the jmap tool, which ships with Sun's Java VM:
jmap -dump:file=<file_name> <pid>

You have to start the Java process with the correct arguments, which vary a little depending on the JVM version. Then, send a QUIT signal to the process to generate a new file.
The output is normally generated when the VM exits, although this can be disabled by setting the “dump on exit” option to “n” (doe=n). In addition, a profile is generated when Ctrl-\ or Ctrl-Break (depending on platform) is pressed. On Solaris OS and Linux a profile is also generated when a QUIT signal is received (kill -QUIT pid). If Ctrl-\ or Ctrl-Break is pressed multiple times, multiple profiles are generated to the one file.

VisualVM can help you dig into what your process is doing, including the ability to arbitrarily force a heap dump on a running process.

jconsole now has the ability to create a dump to the app's current working directory.
Connect to your JMX enabled instance
Navigate to com.sun.management-->HotspotDiagnostic-->Operations
Fill in p0 to name the Heap Dump
press the heapDump button
Jconsole Screen Shot