I have the following problem:
I deploy a web application in Tomcat (Linux) and after shutdown of Tomcat, if I do ps -ef I still can see the java process running.
I believe this happens due to some hanging thread but I don't know how can I track this thread down.
How can I debug this issue?
You can generate 4-5 thread dumps as described below and then analyze them using tools like Samurai.
What you want to check is when a stuck thread or long running transaction happens, all the thread dumps will show a certain thread id is at the same line in your java stack trace. In simpler terms, the transaction is spanning across multiple thread dumps and hence needs more investigation.
Now when you run these through Samurai, it will highlight these in Red colour so you can quickly click on it and get to the lines showing issues.
See an example of this here. Look at the Samurai output image in that link. The Green cells are fine. Red and Grey cells need looking at.
Generating a Thread Dump:
(Linux)
If the JVM is running in a console then simply press Ctrl-\.
If the JVM is running in the background then send it the QUIT signal:
kill -QUIT process_id
There process_id is the process number of the running Java process. The thread dump will be sent to wherever standard out is redirected too.
You can generally get the process numbers of of all running Java processes with the command:
ps axf | grep java
You say your java process still exists, right?
Processes exist as long as they have attached threads, right?
If so, I would go for the following approach:
- run the process with the MBean server attached and managed internally by the JVM
Then connect to the process after you send the QUIT signal and get the thread dump (there should be a JMX for that. See which threads look suspicious to you.
I think you can also use JVisualVM to take thread dumps...
Related
Hadoop jvm process hangs without any error message,
I want to take a look into what JVM processes are doing (where they are stuck).
When I program in C++, I used GDB that can be attached to a running process and show the call stack of the threads.
How can I do the same thing for JVM?
You may use following command
kill -3 [PID]
This will print stack traces of all threads to the console of your java process. Another option is to use jstack utility which is bundled with jdk. Jstack does the same thing.
If it doesn't help then profilers should help. They can gather a lot more data than one thread dump.
I am facing with a strange case. I'd be glad if you could share your comments.
We have solution running on Java 1.6.085 and sometimes Java process is getting hang in production. The solution is running on Linux server.
I investigated GC logs, there is no Full GC. Pause times also look reasonable.
Then we tried to take a thread dump when case happens however kill -3, ./jstack or ./jstack -F do not work. No thread dump could be taken. What could be the reason for that ? Any ideas on investigating the issue ?
BR
-emre
After a while it is understood that the issue occured due to pstack and qdb commands which are executed on java process for operational purposes. Somehow pstack and qdb suspends the java process. Therefore we couldnt be able to take thread or heap dump
We're using jConsole with the topthreads plugin to analyze such cases. The plugin uses JMX to check the thread runtimes and displays their CPU usage since start of the tracking procedure as well as the current stack trace for each thread.
To connect our servers from a local machine we use tunnels in putty, i.e. we first connect to the server via putty and then connect jConsole to a local port which is tunneled to the server.
I have a init.d script which starts/stops a jruby based server running on java 8. It first does it the nice way with a kill and then after a while it falls back to a kill -9. I can see from my logs that the kill triggers the right shutdown hooks and eventually it calls System.exit(0). At this point the process should die, except it doesn't.
I've actually polled with a loop inside my init.d script whether the process still exists at that point (yes) and tried with a kill -QUIT to make it log a thread dump. The latter stops working after a few seconds but the process zombies along until I kill -9 it.
My question: how can I determine what is causing this? At least a thread dump would tell me what part of my code is blocking but it seems to be in a state where that no longer works and yet the process does not exit.
The suggestion by Joe to use Runtime.getRuntime().halt(int) works. Thanks
I am new in Java and have got as a task to find out, why after some time running server takes 100% of the CPU. I think I should look, what the threads are doing in this application. As I wanted to stop the Service, the 5605th thread has been started. How do I control which threads are stopped or not, or run for too long?
Thanks.
If you are on *nix environment first try to find the process id
Get the process ID.
ps -ef | grep "java"
Get the Thread dump.
kill -3 processid
Look into the server logs what threads are there and if there are any "Blocking" threads then they are the culprits look at the stack trace and it may give some clues.
The server and client jvms (Java Virtual Machine) are different. The client thinking is basically "you will be one among other dudes trying to use the same resources". The server thinking is "go ahead, you got all the playground for yourself".
How do you generate and analyze a thread dump from a running JBoss instance?
There is a JBoss-specific method that is slightly more user-friendly:
http://community.jboss.org/wiki/GenerateAThreadDumpWithTheJMXConsole
This is especially useful when you don't have direct access to the host machine (which "kill" would require).
http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/
...
"On UNIX platforms you can send a signal to a program by using the kill command. This is the quit signal, which is handled by the JVM. For example, on Solaris you can use the command kill -QUIT process_id, where process_id is the process number of your Java program.
Alternatively you can enter the key sequence <ctrl>\ in the window where the Java program was started. Sending this signal instructs a signal handler in the JVM, to recursively print out all the information on the threads and monitors inside the JVM."
...
"Determining the Thread States
You will see many different threads in many different states in a snapshot from a JVM stack trace. The key used is:
R Running or runnable thread
S Suspended thread
CW Thread waiting on a condition variable
MW Thread waiting on a monitor lock
MS Thread suspended waiting on a monitor lock"
The stacktrace app found here is also useful, especially on Windows machines when the java app is not started from the command line.
Two options:
OPTION 1 Generate a thread dump using JMX Console
In order to generate a thread dump:
Open the JMXConsole (for example: http://localhost:8080 )
Navigate to jboss.system:type=ServerInfo mbean (hint: you can probably just CTRL-F and enter type=ServerInfo in the dialog box)
Click on the link for the Server Info mbean.
Navigate to the bottom where it says listThreadDump
Click it and get your thread dump
Notes:
If you are using Internet Explorer you should use File > Save As to save the output instead of copying the data to a text editor. For some reason when you copy the text from Internet Explorer the line breaks are not copied and all of the output ends up on a single line.
OPTION 2 Generate a Thread Dump using Twiddle
Alternatively you can use twiddle to execute the listThreadDump() method and pipe the returned HTML directly to file. Use this command line:
<JBOSS_HOME>/bin/twiddle invoke "jboss.system:type=ServerInfo" listThreadDump > threads.html
Thread.getAllStackTraces() (since Java 1.5)
Sometimes JBoss locks so much that even jmx-concole doesn't respond.
In such case use kill -3 on Linux and SendSignal on Windows.
https://community.jboss.org/wiki/ThreadDumpJSP page features standalone self-contained threaddump.war that can be used without JMX.