Hadoop jvm process hangs without any error message,
I want to take a look into what JVM processes are doing (where they are stuck).
When I program in C++, I used GDB that can be attached to a running process and show the call stack of the threads.
How can I do the same thing for JVM?
You may use following command
kill -3 [PID]
This will print stack traces of all threads to the console of your java process. Another option is to use jstack utility which is bundled with jdk. Jstack does the same thing.
If it doesn't help then profilers should help. They can gather a lot more data than one thread dump.
Related
I am trying to monitor a process (which crashed due to Out Of Memory) last time on one production machine. The process is running with -XX:OnOutOfMemoryError=jmap option. What does it mean? Does it mean that it would produce a heap dump on OutOfMemory? or the command jmap is incomplete and should have more to it?
-XX:OnOutOfMemoryError= string is used to specify a command or script to execute when an OutOfMemoryError is first thrown. Documentation is your friend.
I work on very large web project which is written in java.
when I click some button or do other actions it is hard to me to understand what methods called in application code(because I am new in project and application is really really big). So I would like to know is there a tool which will allow to get stacktrace of some threads with given interval (say every 100 milliseconds ).
I know about VisualVm but it does not allow to do this, I can get thread dumb only at one point of time( there is no way to get stack trace continuously).
Can someone suggest tool or any technique which will allow me to monitor methods call at run-time.?
Thanks
For such cases I use Java Mission Control. The full features works on Oracle JDK, for OpenJDK not everything works properly. More info
From the website:
Starting with the release of Oracle JDK 7 Update 40 (7u40), Java Mission Control is bundled with the HotSpot JVM.
You need to add the following parameters in your JVM to be able to use it. note: I normally add also the debug options.
JAVA_DEBUG="-Xdebug -Xrunjdwp:transport=dt_socket,address=4000,server=y,suspend=n"
JAVA_JMC="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=3614 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:+UnlockCommercialFeatures -XX:+FlightRecorder"
Then you'll need to remote attach to port 3614 and you'll be able to see inside the JVM. There you'll be able to profile CPU, check allocation and deadlock detection + select the thread and see what is currently executing. And some other graphs and valuable information.
There are multiple ways in which you could check stacktrace:
Using jconsole's thread tab where you will get to see which all threads are alive and what state they are at.
Using JvisualVm (which comes fee with jdk installation) or you could use any of profilers like jprofiler/yourkit etc. to view stack trace when you run in development mode.
You could get stack trace say every minute by running kill -3 pid in unix or control + break on windows
You could use jstack command to get trace.
You could debug the code using say IDE (eclipse/netbeans/Intellij etc.)and after each and every method call trace the method call.
Many tools for Java VM monitoring and application monitoring exist. For instance, see the eG Java Application Monitor:
...the eG Java Monitor gives you a comprehensive view of the
activities within a JVM:
It lets you see which threads are running in the JVM and what state
they are in (such as runnable, blocked, waiting, timed waiting,
deadlocked, or high CPU).
You also have access to a stack trace for
each thread showing class, method, and line of code (to troubleshoot
problems down to the line of code level).
And you can monitor the
performance of garbage collection processes, CPU and memory usage, and
JVM restarts.
I am analyzing the differences between approaches for taking thread dumps. Below are the couple of them I am researching on
Defining a jmx bean which triggers jstack through Runtime.exec() on clicking a declared bean operation.
Daemon thread executing "ManagementFactory.getThreadMXBean().dumpAllThreads(true, true)" repeatedly after a predefined interval.
Comparing the thread dump outputs between the two, I see the below disadvantages with approach 2
Thread dumps logged with approach 2 cannot be parsed by open source thread dump analyzers like TDA
The ouput does not include the native thread id which could be useful in analyzing high cpu issues (right?)
Any more?
I would appreciate to get suggestions/inputs on
Are there any disadvantages of executing jstack through Runtime.exec() in production code? any compatibility issues on various operating systems - windows, linux?
Any other approach to take thread dumps?
Thank you.
Edit -
A combined approach of 1 and 2 seems to be the way to go. We can have a dedicated thread running in background and printing the thread dumps in the log file in a format understood by the thread dump analyzers.
If any extra information is need (like say probably the native thread id) which is logged only by the jstack output, we do it manually as required.
You can use
jstack {pid} > stack-trace.log
running as the user on the box where the process is running.
If you run this multiple times you can use a diff to see which threads are active more easily.
For analysing the stack traces I use the following sampled periodically in a dedicated thread.
Map<Thread, StackTraceElement[]> allStackTraces = Thread.getAllStackTraces();
Using this information you can obtain the thread's id, run state and compare the stack traces.
With Java 8 in picture, jcmd is the preferred approach.
jcmd <PID> Thread.print
Following is the snippet from Oracle documentation :
The release of JDK 8 introduced Java Mission Control, Java Flight Recorder, and jcmd utility for diagnosing problems with JVM and Java applications. It is suggested to use the latest utility, jcmd instead of the previous jstack utility for enhanced diagnostics and reduced performance overhead.
However, shipping this with the application may be licensing implications which I am not sure.
If its a *nix I'd try kill -3 <PID>, but then you need to know the process id and maybe you don't have access to console?
I'd suggest you do all the heap analysis on a staging environment if there is such an env, then reflect your required Application Server tuning on production if any. If you need the dumps for analysis of your application's memory utilization, then perhaps you should consider profiling it for a better analysis.
Heap dumps are usually generated as a result of OutOfMemoryExceptions resulting from memory leaks and bad memory management.
Check your Application Server's documentation, most modern servers have means for producing dumps at runtime aside from the normal cause I mentioned earlier, the resulting dump might be vendor specific though.
I have the following problem:
I deploy a web application in Tomcat (Linux) and after shutdown of Tomcat, if I do ps -ef I still can see the java process running.
I believe this happens due to some hanging thread but I don't know how can I track this thread down.
How can I debug this issue?
You can generate 4-5 thread dumps as described below and then analyze them using tools like Samurai.
What you want to check is when a stuck thread or long running transaction happens, all the thread dumps will show a certain thread id is at the same line in your java stack trace. In simpler terms, the transaction is spanning across multiple thread dumps and hence needs more investigation.
Now when you run these through Samurai, it will highlight these in Red colour so you can quickly click on it and get to the lines showing issues.
See an example of this here. Look at the Samurai output image in that link. The Green cells are fine. Red and Grey cells need looking at.
Generating a Thread Dump:
(Linux)
If the JVM is running in a console then simply press Ctrl-\.
If the JVM is running in the background then send it the QUIT signal:
kill -QUIT process_id
There process_id is the process number of the running Java process. The thread dump will be sent to wherever standard out is redirected too.
You can generally get the process numbers of of all running Java processes with the command:
ps axf | grep java
You say your java process still exists, right?
Processes exist as long as they have attached threads, right?
If so, I would go for the following approach:
- run the process with the MBean server attached and managed internally by the JVM
Then connect to the process after you send the QUIT signal and get the thread dump (there should be a JMX for that. See which threads look suspicious to you.
I think you can also use JVisualVM to take thread dumps...
When I issue a kill -3 <pid> command to my Java program, it generates the thread dump on the console. How do I redirect this to a file?
Two options:
Run your Java application with stdout redirected
java com.example.MyApp > out.txt
Use jstack instead.
The jstack utility allows you to get a thread dump and send the output to the current console instead of the stdout of the Java application, allowing you to redirect it.
For example, if the PID of your Java application is 12345 (use the jps utility to find it quickly):
jstack 12345 > threads.txt
I usually use the NetBeans profiler, but jvisualvm is available from the command line.
If you want details of all threads and other JVM details, try jconsole.
Please append following JVM arguments to your application. Thread dump should be captured at dump.log.
-XX:+UnlockDiagnosticVMOptions -XX:+LogVMOutput -XX:LogFile=dump.log
Please note it does not redirect, but enables JVM diagnostic logging. So, there could be possible over head as well.
However, if you can have JDK in the environment, using jstack or jcmd (jcmd is preferred with JDK 1.8), you can capture thread dump and redirect to a file.
you can generate java thread dumps using 4 ways excluding the kill -QUIT way.