We're having the classic spring/hibernate/mysql stack running in Tomcat 5.5. Once in a while we're getting a deadlock when the attempt times out to lock a table row. Some kind of deadlock exception is thrown.
The exception is clear and the stack trace indicate what went wrong. But it doesn't show the other thread which is holding the actual lock. Unless I know what that thread is doing it's all just a needle in a haystack.
QUESTION: Is there a way to find the other thread ?
Thanks !
Jan
Try using the following command in MySQL next time you see a deadlock. This should show you the last deadlock.
SHOW INNODB STATUS
Typically when you see a deadlock on your application server the logs show only the victim thread (the one which was rolled back). Since the other thread has completed no exception is thrown. You need to go back to your DB to recreate the transactions.
Once you have a capture from your DB for where the deadlock occured then you can investigate further.
not sure if you've figured it out already but if it's a deadlock, thread dump would be of great help here. Depending on what OS the application is run and on your priviledges to access it, you can generate it in many ways.
on *nix sending QUIT signal to the process ('kill -3 pid') would do the work
using jconsole/jvisualvm has an option to get it
using standard jdk jstack (consider -F -l options) will do the trick
if you are lucky to be on solaris pstack will help a lot
Once you've got it, analyse locked/waiting threads to find a deadlock. You can do it manually or using some existing analyzers that utilize deadlock detection algorithms. Btw jvm has one builtin and it can give you the idea right in the thread dump.
If I can help more just let me know.
good luck.
regards,
baz
if it's a code problem you could try to connect to the running process using jconsole and detect the deadlock.
If you need to find the thread that holds a lock, you can do this in Eclipse through the debug view. Have a look at http://archive.eclipse.org/eclipse/downloads/drops/R-3.1-200506271435/eclipse-news-part2b.html and scroll down to 'Debugging locks and deadlocks'.
The locks owned by a thread as well as the lock a thread is waiting for can both be displayed inline in the Debug view by toggling the Show Monitors menu item in the Debug view drop-down menu. Threads and locks involved in a deadlock are highlighted in red.
Related
During debugging my Java Application in Eclipse, the Debug View shows some threads in red text color and with a lock symbol:
Can anyone explain what this means?
I'm currently looking for (potential) deadlocks in my code. That's why the red threads bother me. But those threads are obviously not involved in a deadlock situation, when I suspend the VM:
I studied the Eclipse help pages for Debug View, but found no explanation there.
EDIT: as #howlger correctly points out, there is a note on the Eclipse Tips and Tricks (JDT) page, stating "... Threads involved in a deadlock are rendered in red. ...". But does that also mean deadlock detection is the only reason for rendering the threads in red? If so, my case might just be a false positive guess by the eclipse internal heuristics. As soon as I suspend my VM, eclipse investigates monitor ownership in full detail and finds that there is no actual deadlock situation. This also aligns with the suspended thread appearance (my second screenshot, above): the threads are suddenly not red anymore, when suspended.
The threads in red are involved in a deadlock, but you have to enable Show Monitors to see also which tread owns which object.
See the Eclipse Tips and Tricks (JDT) help page:
Threads and monitors
The Java debugger optionally displays monitor information in the
Debug view. Use the Show Monitors action in the Debug view drop down menu to show which threads are holding locks and which are
waiting to acquire locks. Threads involved in a deadlock are rendered
in red.
I've a java process running and unfortunately one thread inside the process is hung.
I found the Thread id which was hung using jstack, however I was unable to find any references on how to interrupt this thread using the id?
Is it possible to Interrupt/Stop a thread from console (or basically outside the process) by using the processId and ThreadId?
Any suggestions on how to tackle this?
PS : I don't want to kill the process as its just one thread which is hung. Also, neither do I want to make code changes to Stop/Interrupt the thread. I just want to kill it, so all its resources can be released.
There's no baked-in way to kill a thread within the JVM, at least not a deliberately implemented one.
Having said that, if you have started your JVM with the appropriate parameters, so that you can start a remote JMX session to it, you can actually suspend the thread and inject a RuntimeException into it, which will almost surely terminate it (unless you are doing something gnarly with RuntimeExceptions in it).
See this blogpost.
P.S. You would never start your JVM in production allowing rogue JMX connections though, and if you're not in production, I'd guess that the above approach is not of much help to you.
My application run some complex threads that fetch maps in a background thread and draw them. Sometimes if I run the app for a couple hours on a slow network I seem to be getting it into a weird state where all my threads status are showing TimedWait or Wait (except the ones that are Native such as main).
What is the cause of this? How can I debug it? I am absolutely lost and I know this is a bit of a general question but I would appreciate it if someone could point me to the right direction. EG:
How to pin point the cause of the problem.
What king of issues generally cause all the threads to lock up?
Anybody seen anything similar?
Thanks
A timed wait is simply a thread which is blocked on some O/S level call which has a timeout specified, such as a simple wait primitive (Object.wait()) or a socket operation (Socket read()/write()), a thread queue etc. It's quite normal for any complex program to have several or many of these - I have an application server which routinely has hundreds, even thousands.
Your threads may be backing up on non-responsive connections and may not be misbehaving at all, per se. It may simply be that you need to program them to detect and abort an idle connection.
Click on each of the threads which you are concerned about and analyze their stack trace for how they got there.
Most decent profiling tools (and application containers) will have the option of printing a full stack trace, and more modern ones will do a dead-lock and live-lock analysis for you. The JVisualVM tool distributed with Sun's JDK and available on the net as VisualVM will do this and it's very effective. Most decent profilers will also show lock acquisition in the stack trace (yours, above, is not in that view).
Otherwise, you are looking for two or more threads contending for the same lock or acquiring the same locks in a different order. You may need to do this manually by actually examining the source and annotating your stack trace, but you should be able to whittle down likely candidates if your tool doesn't point right to the conflicting threads.
I am trying to debug a program in Java using SpringSource ToolSuite. Every time it gets stuck at AWTAutoShutdown.class. I have tried hitting F8 to step out of whatever is happening but it seems like there is a thread locking up somewhere. I understand this is kind of a vague question but has anyone seen this before? Where can I start looking to solve this problem?
So one thread is suspended with a ThreadDeath exception. Perhaps STS is suspending because there is an uncaught exception, instead of on a breakpoint. There is a setting so that you can turn off suspension on uncaught exceptions, which I only want about half the time anyway. Turn that off and see if your thread dies and gives you something more useful in terms of operation (or an error).
I have a multithreaded application. Several messages are coming to the application and are processed in separated threads. For this I am using classes ThreadPoolExecutor and FutureTask from package java.util.concurrent.
Occasionally I have some deadlocks in the application. When a deadlock occurs I want to interrupt the blocking thread and I want to log the stack trace of this thread so that I can later resolve the deadlock.
Is there any way how can we find the stack trace of a thread outside of that thread in Java?
See here for how to generate stack traces, including how to do this programatically. From the console, Ctrl+Break will dump the stack traces to stdout. See also this SO question for more details.
You could log the stack traces of all thread from time to time (or before killing the process) from within your application. To do that use:
Map<Thread, StackTraceElement[]> m = Thread.getAllStackTraces();
for(Map.Entry<Thread, StackTraceElement[]> e : m.entrySet()) {
log(e.getKey().toString());
for (StackTraceElement s : e.getValue()) {
log(" " + s);
}
}
When running nightly automated tests, sometimes some one of the test cases gets into a deadlock. I added a "TimeBomb" daemon thread that waits 30 minutes, and if then logs all stack traces as above.
Before entering the deadlock region, set a field like,
thread = Thread.currentThread();
In your monitoring thread you can perform thread.getStackTrace(); to get the stack trace of that thread at any time.
You use JStack. Here is a nice blog entry that details how to get stack traces.
I wasn't sure if you wish to obtain the stacktrace from within the same JVM or externally, but if you wish to obtain the stack trace with external tools, the following will help:
The Java VisualVM tool can be used to connect to the running JVM, where the thread stack can be dumped. This is usually the preferred approach for most people using Java 6. Once VisualVM is launched, the thread dump of the process can be obtained by selecting the process in the Application tab. A Threads tab is now made available to view the threads in the running process, and in the same tab you'll find the "Thread Dump" button to extract the required information.
The jstack utility within the JDK can also be used to produce thread stacktraces.
When a deadlock occurs I want to interrupt the blocking thread ...
You could implement a periodic task to check for deadlocks (where deadlocks are java intrinsic or Lock based) and call interrupt on all threads involved in the scenario. However, this has no guarantees that it will solve your problem. Its likely the scenario that will just happen again. See Dr Heinz's article on a deadlock detector for details.
If fact, there is no guarantee that interrupt will even free up a blocked process like this. Its a far better approach to avoid the deadlock scenario in the first place by, for example, using locks with timeouts and retry strategies or 'try before you buy' approaches.
and I want to log the stack trace of this thread...
If you want to do this programatically, again, follow Dr Heinz's example. If not, just generate the thread dump when you've spotted the problem.
Is there any way how can we find the stack trace of a thread outside of that thread in Java?
Yes and no. You can dump the threads from other VMs but their stack traces may not be as useful as you might think to determining the causes of your deadlock. If a genuine deadlock has been detected (by the JVM itself on thread dump of your applications VM) you should have everything you need to debug the cause (more or less).