I have a multithreaded application. Several messages are coming to the application and are processed in separated threads. For this I am using classes ThreadPoolExecutor and FutureTask from package java.util.concurrent.
Occasionally I have some deadlocks in the application. When a deadlock occurs I want to interrupt the blocking thread and I want to log the stack trace of this thread so that I can later resolve the deadlock.
Is there any way how can we find the stack trace of a thread outside of that thread in Java?
See here for how to generate stack traces, including how to do this programatically. From the console, Ctrl+Break will dump the stack traces to stdout. See also this SO question for more details.
You could log the stack traces of all thread from time to time (or before killing the process) from within your application. To do that use:
Map<Thread, StackTraceElement[]> m = Thread.getAllStackTraces();
for(Map.Entry<Thread, StackTraceElement[]> e : m.entrySet()) {
log(e.getKey().toString());
for (StackTraceElement s : e.getValue()) {
log(" " + s);
}
}
When running nightly automated tests, sometimes some one of the test cases gets into a deadlock. I added a "TimeBomb" daemon thread that waits 30 minutes, and if then logs all stack traces as above.
Before entering the deadlock region, set a field like,
thread = Thread.currentThread();
In your monitoring thread you can perform thread.getStackTrace(); to get the stack trace of that thread at any time.
You use JStack. Here is a nice blog entry that details how to get stack traces.
I wasn't sure if you wish to obtain the stacktrace from within the same JVM or externally, but if you wish to obtain the stack trace with external tools, the following will help:
The Java VisualVM tool can be used to connect to the running JVM, where the thread stack can be dumped. This is usually the preferred approach for most people using Java 6. Once VisualVM is launched, the thread dump of the process can be obtained by selecting the process in the Application tab. A Threads tab is now made available to view the threads in the running process, and in the same tab you'll find the "Thread Dump" button to extract the required information.
The jstack utility within the JDK can also be used to produce thread stacktraces.
When a deadlock occurs I want to interrupt the blocking thread ...
You could implement a periodic task to check for deadlocks (where deadlocks are java intrinsic or Lock based) and call interrupt on all threads involved in the scenario. However, this has no guarantees that it will solve your problem. Its likely the scenario that will just happen again. See Dr Heinz's article on a deadlock detector for details.
If fact, there is no guarantee that interrupt will even free up a blocked process like this. Its a far better approach to avoid the deadlock scenario in the first place by, for example, using locks with timeouts and retry strategies or 'try before you buy' approaches.
and I want to log the stack trace of this thread...
If you want to do this programatically, again, follow Dr Heinz's example. If not, just generate the thread dump when you've spotted the problem.
Is there any way how can we find the stack trace of a thread outside of that thread in Java?
Yes and no. You can dump the threads from other VMs but their stack traces may not be as useful as you might think to determining the causes of your deadlock. If a genuine deadlock has been detected (by the JVM itself on thread dump of your applications VM) you should have everything you need to debug the cause (more or less).
Related
My application run some complex threads that fetch maps in a background thread and draw them. Sometimes if I run the app for a couple hours on a slow network I seem to be getting it into a weird state where all my threads status are showing TimedWait or Wait (except the ones that are Native such as main).
What is the cause of this? How can I debug it? I am absolutely lost and I know this is a bit of a general question but I would appreciate it if someone could point me to the right direction. EG:
How to pin point the cause of the problem.
What king of issues generally cause all the threads to lock up?
Anybody seen anything similar?
Thanks
A timed wait is simply a thread which is blocked on some O/S level call which has a timeout specified, such as a simple wait primitive (Object.wait()) or a socket operation (Socket read()/write()), a thread queue etc. It's quite normal for any complex program to have several or many of these - I have an application server which routinely has hundreds, even thousands.
Your threads may be backing up on non-responsive connections and may not be misbehaving at all, per se. It may simply be that you need to program them to detect and abort an idle connection.
Click on each of the threads which you are concerned about and analyze their stack trace for how they got there.
Most decent profiling tools (and application containers) will have the option of printing a full stack trace, and more modern ones will do a dead-lock and live-lock analysis for you. The JVisualVM tool distributed with Sun's JDK and available on the net as VisualVM will do this and it's very effective. Most decent profilers will also show lock acquisition in the stack trace (yours, above, is not in that view).
Otherwise, you are looking for two or more threads contending for the same lock or acquiring the same locks in a different order. You may need to do this manually by actually examining the source and annotating your stack trace, but you should be able to whittle down likely candidates if your tool doesn't point right to the conflicting threads.
I am using samurai tool to analyze thread dump. It looks like it has many blocked threads. I have no clue to derive anything from the thread dump.
I have an SQL query in my Java application that runs on weblogic that takes enormous time to complete. After running this query by clicking on my Java application button several times hangs my JVM.
Thread dumps can be found # : http://www.megafileupload.com/en/file/379103/biserver2-txt.html
Can you help me understand what does the thread dump say ?
The amount of data you provide is a bit overwhelming, so let's just give you a hint how to proceed. For the analysis I use open source threadlogic application based on TDA. It takes few seconds to parse 3 MiB worth of data but in nicely shows 22 different stack trace dumps in one file:
Drilling down to reveals really disturbing list of warnings and alerts.
I don't have time to examine all of them, but here is a list of those marked as FATAL (keep in mind that false-positives are also to be expected):
Wait for SLSB Beans
Description: Waiting for Stateless Session Bean (SLSB) instance from the SLSB Free pool
Advice: Beans all in use, free pool size size insufficient
DEADLOCK
Description: Circular Lock Dependency Detected leading to Deadlock
Advice: Deadlock detected with circular dependency in locks, blocked threads will not recover without Server Restart. Fix the order of locking and or try to avoid locks or change order of locking at code level, Report with SR for Server/Product Code
Finalizer Thread Blocked
Description: Finalizer Thread Blocked
Advice: Check if the Finalizer Thread is blocked for a lock which can lead to wasted memory waiting to be reclaimed from Finalizer Queue
WLS Unicast Clustering unhealthy
Description: Unicast messaging among Cluster members is not healthy
Advice: Unicast group members are unable to communicate properly, apply latest Unicast related patches and enable Message Ordering or switch to Multicast
WLS Muxer is processing server requests
Description: WLS Muxer is handling subsystem requests
Advice: WLS Server health is unhealthy as some subsystems are overwhelmed with requests which is leading to the Muxer threads directly handling requests. instead of dispatching to relevant subsystems. There is likely a bug here.
Stuck Thread
Description: Thread is Stuck, request taking very long time to finish
Advice: Check why the thread or call is taking very long??. Is it blocked for unavailable or bad resource or contending for Lock?. Can be ignored if it is doing repeat work in a loop. (like adapter threads polling for events in a infinite loop)...
The issue was with WLDF logging information to log file. Once disabled it helped improve performance enormously. I am not a fan of ThreadLogic as a tool for thread dump analysis. It shows circular deadlock when you have stuck threads no matter how variant the issue is.
Thread dumps are the snapshot of all threads running in the application at given moment. Thread dump will have hundreds/thousands of application threads. It would be hard to scroll through every single line of the stack trace in every single thread. Call Stack Tree consolidates all the threads stack trace into one single tree and gives you one single view. It makes the thread dumps navigation much simpler and easier. Below is the sample call stack tree generated by fastThread.io.
Fig 1: Call stack Tree
You can keep drilling down to see code execution path. Fig 2 shows the drilled down version of a particular branch in the Call Stack Tree diagram.
Fig 2: Drilled down Call Stack Tree
Sample call stack tree generated by FastThread.io
I want to know in every moment of program execution what thread is executed now (not all threads in JVM but only threads that belong to my program).
How can I get it?
YourKit can show you which threads were running, blocked, waiting etc at any given moment and give you a snapshot of the stack at intervals.
You can create a process to poll all the threads yourself with Thread.getAllStackTraces() which gives you the stack trace of every thread. Using a GUI tool is much, much easier.
I sometimes have to look at thread dumps from a Tomcat server. However, this is a very slow process as my application uses thread pools with a couple of hundred threads. Most of the thread dumps I look at include the same stack trace for many of the threads as they are idle waiting for work.
Are there any tools which would parse a thread dump and only show me the unique stack traces along with a count of the number of threads in each state? This would allow me to quickly ignore the tens or hundreds of threads which are waiting in a common location for work.
I have tried the Thread Dump Analyzer but this doesn't do any summarisation of common stack traces.
I have written a tool to do what I wanted.
Java Thread Dump Analysis Tool
Haven't used it for a while but Samurai might be of interest.
Here's an online tool that does exactly what you ask for, no installation necessary:
http://spotify.github.io/threaddump-analyzer/
We're having the classic spring/hibernate/mysql stack running in Tomcat 5.5. Once in a while we're getting a deadlock when the attempt times out to lock a table row. Some kind of deadlock exception is thrown.
The exception is clear and the stack trace indicate what went wrong. But it doesn't show the other thread which is holding the actual lock. Unless I know what that thread is doing it's all just a needle in a haystack.
QUESTION: Is there a way to find the other thread ?
Thanks !
Jan
Try using the following command in MySQL next time you see a deadlock. This should show you the last deadlock.
SHOW INNODB STATUS
Typically when you see a deadlock on your application server the logs show only the victim thread (the one which was rolled back). Since the other thread has completed no exception is thrown. You need to go back to your DB to recreate the transactions.
Once you have a capture from your DB for where the deadlock occured then you can investigate further.
not sure if you've figured it out already but if it's a deadlock, thread dump would be of great help here. Depending on what OS the application is run and on your priviledges to access it, you can generate it in many ways.
on *nix sending QUIT signal to the process ('kill -3 pid') would do the work
using jconsole/jvisualvm has an option to get it
using standard jdk jstack (consider -F -l options) will do the trick
if you are lucky to be on solaris pstack will help a lot
Once you've got it, analyse locked/waiting threads to find a deadlock. You can do it manually or using some existing analyzers that utilize deadlock detection algorithms. Btw jvm has one builtin and it can give you the idea right in the thread dump.
If I can help more just let me know.
good luck.
regards,
baz
if it's a code problem you could try to connect to the running process using jconsole and detect the deadlock.
If you need to find the thread that holds a lock, you can do this in Eclipse through the debug view. Have a look at http://archive.eclipse.org/eclipse/downloads/drops/R-3.1-200506271435/eclipse-news-part2b.html and scroll down to 'Debugging locks and deadlocks'.
The locks owned by a thread as well as the lock a thread is waiting for can both be displayed inline in the Debug view by toggling the Show Monitors menu item in the Debug view drop-down menu. Threads and locks involved in a deadlock are highlighted in red.