I want to get the thread dump of my web app that running on a jboss server.
I found two solutions for my problem :
Using the unix command : kill -3
Using the jstack tool that exists in the JDK.
Can anyone explain to me the difference between theses two methods?
Thanks in advance !
The jstack command can get a thread dump of a program running on a remote machine, and it also works on Windows.
kill -3 only works on local programs, and on Windows there is no kill.
From the oracle page of jstack:
The output from the jstack pid option is the same as that obtained by pressing Ctrl+\ at the application console (standard input) or by sending the process a QUIT signal.
Also remember that Ctrl+\ is equivalent to a SIGQUIT.
From what is kill -3 (unix.se):
kill -l shows us all signals. Following this hint 3 means SIGQUIT
So basically both of them do exactly the same thing, i.e asking for a coredump. Here are some pointers related to jstack:
Jstack performs deadlock detection by default.
Regarding official support, from the jstack man page:
Prints Java thread stack traces for a Java process, core file, or remote debug server. This command is experimental and unsupported.
This utility is unsupported and might not be available in future release of the JDK. In Windows Systems where the dbgeng.dll file is not present, Debugging Tools For Windows must be installed so these tools work.
Regarding the output difference, its basically the same thing. There is a one to one mapping between the outputs. See my output for the same application to demonstrate the mapping between the statuses of kill -3 and jstack. The mapping between the statuses are:
kill -3 | Jstack
------------------------------
RUNNABLE | IN_NATIVE
TIMED_WAITING | BLOCKED
WAITING | BLOCKED (PARK)
In Windows you have something called "taskkill /PID {yourpid} /F" for killin process. The process id can be obtained from netstat command or use viusal vm to know process id
Related
I would like to automatically create a thread dump as part of a log collection script I have written.
I know that it is possible to generate a thread dump using jstack or kill -3 . The customer running the log collection script will only have a JRE installed so jstack is not an option.
If I use jstack and the pid of the JVM that the install4j launcher creates I get the thread dump...however invoking kill -3 with the same pid generates no output. I am currently directing both stderror and stdout to files using the install4j launcher configuration.
This is a linux launcher that is configured with the service option.
FWIW...I have tried not directing the output and also running my application as a console program instead of a service and none of these allow 'kill -3' to work.
I'm using install4j 6.0
Lastly, I do not want to use -XX:+UnlockDiagnosticVMOptions -XX:+LogVMOutput -XX:LogFile=dump.log because of the unknown overhead it might create.
Any help would be appreciated!
I would suggest to use the ThreadMXBean to get this information programmatically in the same process rather than use signal handlers from another process. The API documentation is at
http://docs.oracle.com/javase/8/docs/api/java/lang/management/ThreadMXBean.html
Here's a full example of how to use this MBean.
In that way you can write the information explicitly to wherever you want it to go.
I have the following problem:
I deploy a web application in Tomcat (Linux) and after shutdown of Tomcat, if I do ps -ef I still can see the java process running.
I believe this happens due to some hanging thread but I don't know how can I track this thread down.
How can I debug this issue?
You can generate 4-5 thread dumps as described below and then analyze them using tools like Samurai.
What you want to check is when a stuck thread or long running transaction happens, all the thread dumps will show a certain thread id is at the same line in your java stack trace. In simpler terms, the transaction is spanning across multiple thread dumps and hence needs more investigation.
Now when you run these through Samurai, it will highlight these in Red colour so you can quickly click on it and get to the lines showing issues.
See an example of this here. Look at the Samurai output image in that link. The Green cells are fine. Red and Grey cells need looking at.
Generating a Thread Dump:
(Linux)
If the JVM is running in a console then simply press Ctrl-\.
If the JVM is running in the background then send it the QUIT signal:
kill -QUIT process_id
There process_id is the process number of the running Java process. The thread dump will be sent to wherever standard out is redirected too.
You can generally get the process numbers of of all running Java processes with the command:
ps axf | grep java
You say your java process still exists, right?
Processes exist as long as they have attached threads, right?
If so, I would go for the following approach:
- run the process with the MBean server attached and managed internally by the JVM
Then connect to the process after you send the QUIT signal and get the thread dump (there should be a JMX for that. See which threads look suspicious to you.
I think you can also use JVisualVM to take thread dumps...
I'm running a daemon java process on my ubuntu machine :
java -cp (...) &> err.log&
The process runs for a random period of time and then just disappears. Nothing in LOGs, err.log, no JVM crash file created (hs_err_*.log), nothing. My two questions are :
1) Under which circumstances can a java process abruptly finish ?
2) IS there any way to know what happened to the process (knowing PID) ? Does UNIX keep information about finished processes somehow ?
1) Under which circumstances can a java process abruptly finish ?
When it exits by its own but I guess you ruled out that or when it is killed with SIGKILL. This might be the oom killer if you are on Linux. Did you look at the system message logs ?
2) IS there any way to know what happened to the process (knowing PID) ?
Generally not unless you configure some tracing tools to get that information
Does UNIX keep information about finished processes somehow ?
No, but depending on the Unix variant you are using, that might be something simple to add.
In your example, you can just print the process exit status with echo $?
If it is 265, that would mean the process was killed with signal 9 (=265-256).
I would write a simple shell script that somehow alerts me when the JVM terminated. Perhaps send an email with the JVM's exit code.
#!/bin/sh
# Launch JVM and wait...
java -cp ...
# do something with the exit code $?
# log it to a file or mail it yourself
Perhaps, the exit code might reveal something.
I would run it as a daemon with YAJSW as it allows several ways to monitor the memory etc, has restart options, and you can also enable log on the wrapper process so you can have much info when there is an issue.
When I issue a kill -3 <pid> command to my Java program, it generates the thread dump on the console. How do I redirect this to a file?
Two options:
Run your Java application with stdout redirected
java com.example.MyApp > out.txt
Use jstack instead.
The jstack utility allows you to get a thread dump and send the output to the current console instead of the stdout of the Java application, allowing you to redirect it.
For example, if the PID of your Java application is 12345 (use the jps utility to find it quickly):
jstack 12345 > threads.txt
I usually use the NetBeans profiler, but jvisualvm is available from the command line.
If you want details of all threads and other JVM details, try jconsole.
Please append following JVM arguments to your application. Thread dump should be captured at dump.log.
-XX:+UnlockDiagnosticVMOptions -XX:+LogVMOutput -XX:LogFile=dump.log
Please note it does not redirect, but enables JVM diagnostic logging. So, there could be possible over head as well.
However, if you can have JDK in the environment, using jstack or jcmd (jcmd is preferred with JDK 1.8), you can capture thread dump and redirect to a file.
you can generate java thread dumps using 4 ways excluding the kill -QUIT way.
How do you generate and analyze a thread dump from a running JBoss instance?
There is a JBoss-specific method that is slightly more user-friendly:
http://community.jboss.org/wiki/GenerateAThreadDumpWithTheJMXConsole
This is especially useful when you don't have direct access to the host machine (which "kill" would require).
http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/
...
"On UNIX platforms you can send a signal to a program by using the kill command. This is the quit signal, which is handled by the JVM. For example, on Solaris you can use the command kill -QUIT process_id, where process_id is the process number of your Java program.
Alternatively you can enter the key sequence <ctrl>\ in the window where the Java program was started. Sending this signal instructs a signal handler in the JVM, to recursively print out all the information on the threads and monitors inside the JVM."
...
"Determining the Thread States
You will see many different threads in many different states in a snapshot from a JVM stack trace. The key used is:
R Running or runnable thread
S Suspended thread
CW Thread waiting on a condition variable
MW Thread waiting on a monitor lock
MS Thread suspended waiting on a monitor lock"
The stacktrace app found here is also useful, especially on Windows machines when the java app is not started from the command line.
Two options:
OPTION 1 Generate a thread dump using JMX Console
In order to generate a thread dump:
Open the JMXConsole (for example: http://localhost:8080 )
Navigate to jboss.system:type=ServerInfo mbean (hint: you can probably just CTRL-F and enter type=ServerInfo in the dialog box)
Click on the link for the Server Info mbean.
Navigate to the bottom where it says listThreadDump
Click it and get your thread dump
Notes:
If you are using Internet Explorer you should use File > Save As to save the output instead of copying the data to a text editor. For some reason when you copy the text from Internet Explorer the line breaks are not copied and all of the output ends up on a single line.
OPTION 2 Generate a Thread Dump using Twiddle
Alternatively you can use twiddle to execute the listThreadDump() method and pipe the returned HTML directly to file. Use this command line:
<JBOSS_HOME>/bin/twiddle invoke "jboss.system:type=ServerInfo" listThreadDump > threads.html
Thread.getAllStackTraces() (since Java 1.5)
Sometimes JBoss locks so much that even jmx-concole doesn't respond.
In such case use kill -3 on Linux and SendSignal on Windows.
https://community.jboss.org/wiki/ThreadDumpJSP page features standalone self-contained threaddump.war that can be used without JMX.