I've got a java application that half the time just hangs, and the other half the JVM crashes. Is there a tool I can use to see what's going on that makes it hang and/or crash?
I'm using CentOS 5.6
For starters I would suggest JVisualVM. It comes with the JDK, so you should just need to type jvisualvm into the command line to start it.
Once it starts, you can connect to a running JVM, so you should be able to connect to your hung Java process and inspect the stack dump for all its running threads as well as the contents of the heap.
Other useful built-in tools include:
jps lists process ids of running java processes
jstack prints a stack dump for each thread in the specified JVM process
jmap generates a heap dump for the specified JVM process (jvisualvm can also generate heap dumps)
jhat analyzes heap dumps generated with jmap or jvisualvm
Of couse, there are also more sophisticated profilers available. JProfiler is quite highly regarded.
There are two different cases.
Application crash:
Was that an OOM? NPE? What was the exception? If there was jvm crash you will see hs_err_.log (http://java.sun.com/j2se/1.5/pdf/jdk50_ts_guide.pdf)
Looking at the file you may see if your own JNI caused a crash or JVM bug.
Application Hang: I would start with visualvm or jstat (both are part of JDK). You can see current state of threads and check if there is any application error..
Other linux tools that could help to see inside process:
lsof : you can check if the process opened too many files
strace: see current activity from system call point of view.
Oracle tools documentation provides pretty neat listing. It also links Operating System Specific tools
In these cases(hang, freeze, ...) you have to analyze an heap dump to try to figure out what's happening in your application , you can use JVisualVM to take the dump, or you can add the appropriate JVM parameter to dump the content of the heap in the case of a crash.
Related
I have a Java web server running as a Windows service.
I use Tomcat 8 with Java 1.8.*
For a few months now, I've detected that the memory usage is increasing quite rapidly. I cannot make up for sure if it's heap or stack.
The process starts with ~200MB and after a week or so, it can reach up to 2GB.
Shortly after it will generate OutOfMemory exception (the memory usage will be 2GB - 2.5GB).
This has repeated multiple times on multiple environments.
I would like to know if there's a way to monitor the process and view it's internal memory usage, even to the level of viewing which objects are using the most amount of memory.
Can 'Java Native Memory Tracking' be used for this?
This will help me to detect any memory leaks that might cause this.
Thanks in advance.
To monitor the memory usage of a Java process, I'd use a JMX client such as JVisualVM, which is bundled with the Oracle JDK:
https://visualvm.java.net/jmx_connections.html
To identify the cause of a memory leak, I'd instruct the JVM to take a heap dump when it runs out of memory (on the Oracle JVM, this can be accomplished by specifying -XX:-HeapDumpOnOutOfMemoryError when starting your Java program), and then analyze that heap dump using a tool such as Eclipse MAT.
quoting:
the process starts with ~200MB and after a week or so, it can reach up to 2GB. Shortly after it will generate OutOfMemory exception (the memory usage will be 2GB - 2.5GB).
The problem might not be as simple as seeing what java objects you have got in JVisualVM (e.g millions of strings)
What you need to do is identify the code that leaks.
One way you could do that is to force the execution of particular code and then monitor the memory.
The easiest way to force the execution of code inside classes/objects is to use a tool like https://github.com/lorenzoongithub/nudge4j (particularly since you are on java 8)
alternatively you could just wire up nashorn to a command line or run your progam via jjs https://docs.oracle.com/javase/8/docs/technotes/guides/scripting/nashorn/shell.html
I have Java application, which, unfortunately, begins to consume quite big amounts of memory after some time. To complicate things, it's not only Java application, it is also JavaFX 2 application.
I suspect that there is some memory leak, maybe even in underlying JavaFX calls and native libs.
The ideal solution would be to get a dump of all java objects at some moment (with their memory usage), and then analyze that dump. Is there some way to achieve this?
Use jmap -heap:format=b <process-id> to create a binary dump of the heap which can then be loaded into several tools - my favorite being the "Eclipse Memory Analyzer"
There are lots of ways to get a heap dump, starting with simple tools like jmap to more fancy stuff like JVisualVM or even commerical tools as JProfiler. Correctly interpreting those dumps can be tricky though, so you might want to post exactly what you are looking for. Are going hunting for a memory leak, or are you interested in getting a general feel for your application?
You can use jvisualvm. It has plugin to see live memory and get a dump out of it.
I just re-discovered this article (archive.org archive) when researching ways to grab "JVM state right at this moment" - after a heap I pulled with jmap was about half the size of what the MBeans reported. I'll add it for completeness:
su $JVM_OWNER -c "gcore -o /tmp/jvm.core $YOUR_JVM_PID"
su $JVM_OWNER -c "jmap -dump:format=b,file=jvm.hprof /usr/bin/java /tmp/jvm.core"
Requires gdb installed (for gcore) and a JDK installation (for jmap). Also note you'd might need to adjust /usr/bin/java to the path of the JVM used for the process.
I am running load against Tomcat 6 running on Java 6. I want to collect a heapdump of the Java heap while the Tomcat server is under load. I normally use jmap -dump to collect my heapdumps.
However, when I try to do this when Tomcat is handling a high load I find that the heapdump collection fails.
Is jmap the best tool for collecting a heap dump from a process under load? What are the possible causes which would cause jmap to fail to collect a heapdump?
If jmap is not the best tool - what is better?
It is entirely acceptable to me for jmap (or some other tool) to stop the world within the Java process while the heap dump is taken.
Is jmap the best tool for collecting a heap dump from a process under load?
I think: No it isn't. From this link:
NOTE - This utility is unsupported and
may or may not be available in future
versions of the JDK.
I've also found jmap can pretty temperamental. If you're having problems:
Try it again. It often manages to get a heap dump after a couple of attempts if it first fails
Use the -F option
Add -XX:+HeapDumpOnOutOfMemoryError as a standard configuration to proactively take heap dumps when an OOM error is thrown
Run Tomcat interactively and add the heap dump on ctrl-break option. This gives you a thread dump too, something you'll probably need anyway
If your heap size is especially large and you have a repeatable condition, temporarily lower your heap size. It makes the resulting file much easier to handle, takes less time and is more likely to succeed
I have found that running Tomcat with a JMX port allows me to take a remote heapdump using visualvm. This succeeded for me when jmap failed.
I have java application that is crashing while in production. It doesn't do so in dev/QA. The jvm is creating a .mdmp file and a text file. How do I analyze the binary dump file? I googled but had no luck. We are using bea jrockit jvm 1.5 R27.
The .mdmp file is a Windows MiniDump file that you can only read with a debugger (like WinDbg). Typically you need the sources of the crashed application to really get some information out of the dump. So in your case you can't do much but contacting JRockit support.
Here a link to the Orace JRockit information about JVM crahes.
.mdmp files are the Windows equivalent of unix/linux core dumps. You can analyse them with WinDBG but if it's a Java process that has crashed most likely you'll want to use Java's own tools to analyse the crashed process.
If you want to look at the heap of the crashed Java process you can use a tool that ships with the JDK called jmap to extract a HPROF file from a .core or .mdmp and then load this into a memory analyser. Note also that some memory analyzers can load core dumps and Windows minidumps directly.
Related issue and the jmap docs
If you want to see the state of the threads then you can use a tool called jstack to print stack traces for every thread at the point the dump was created. jstack docs.
So far I have learned about generating thread dump and heap dump using jstack and and jmap respectively.
However, jstack thread dump contains only texts describing the stack on each thread. And opening heap dump (.hprof file) with Java VisualVM only shows the objects allocated in the heap.
What I actually want is to be able see the stack, to switch to particular stack frame, and watch local variables. This kind of post-mortem debugging can be done normally with tools like WinDbg, gdb and a core file (for a native C++ program.)
I wonder if such 'core' file (which will allow me to debug in non-live environment) exists in Java?
Java does. If you are using an IBM VM, use com.ibm.jvm.Dump.SystemDump() to programatically generate a dump. This can be debugged using a debugger. I believe "kill"ing your Java process should generate a system dump too. For Unix use kill -4 pid where pid is the process id and could be queried by typing in top | grep java if you have 1 VM instance running.
You could also add -Xdump:system or -Xdump:heap etc to your java command line to filter events and generate dumps on certain events like VM Stop (-Xdump:system:events=vmstop), full garbage collections(-Xdump:system:events=fullgc), etc. Note, depending on your heap size, generating a dump on a full GC is may not be a good idea (i.e you might create 50 dumps withing 20 seconds if you heap grows from 4M to around 60M in 20 seconds ) so you could add a counter like -Xdump:system:events=fullgc,range=50..55 which would generate 5 cores between the 50th to the 55th full garbage collect.
I've found relevant information in a Sun forum and in an SO discussion: I have not had much luck with it, but it might work in your case.
Note: some of the tools mentioned are Java tools, but are unsupported and are not available on Windows versions of the JDK.
I don't think such a dump mechanism exists in standard Java.
Some operating systems (for example Solaris mdb or gdb on Linux) support using the normal native debugger on dump files, with some special support for showing Java stack frames. But this is pretty hardcore and probably not what you want, since it is not well integrated with the Java Debugger.