I sometimes have to look at thread dumps from a Tomcat server. However, this is a very slow process as my application uses thread pools with a couple of hundred threads. Most of the thread dumps I look at include the same stack trace for many of the threads as they are idle waiting for work.
Are there any tools which would parse a thread dump and only show me the unique stack traces along with a count of the number of threads in each state? This would allow me to quickly ignore the tens or hundreds of threads which are waiting in a common location for work.
I have tried the Thread Dump Analyzer but this doesn't do any summarisation of common stack traces.
I have written a tool to do what I wanted.
Java Thread Dump Analysis Tool
Haven't used it for a while but Samurai might be of interest.
Here's an online tool that does exactly what you ask for, no installation necessary:
http://spotify.github.io/threaddump-analyzer/
Related
I was reading about the concepts of Thread Dumps and found various ways to take Thread Dumps but no articles have mentioned on what basis/issues/reasons the dumps should be taken. We perform a load test on a particular server targeting a JVM and take the Thread Dumps when we observe high CPU utilization or hogging threads. Is this correct ? Can someone throw some light on the reasons on when Thread Dumps should be taken normally or during any load tests.
We use the jstack command to capture the dump:
/app/jdk/jdk1.7.0_111/bin/jstack -l <ProcessID> > <PathToSaveTheFile>
TIA.
Thread dumps are used for post-mortem debugging. IMO, what you are doing is right. I don't see a reason for taking a dump in normal conditions.
Non- Less invasive debugging
You can do thread dumps whenever you see fit to analyse anything to do with thread lock contention, deadlock detection, system resource contention, ...
This is why there are tools to facilitate thread dump whenever we see fit, not only after a JVM crash. Analysis of multiple dumps over time paints fuller picture, than the last crash dump.
This means you can perform less invasive thread debugging without attaching a profiler, which in most cases slows down (and possibly alters some dynamic properties) of the application execution.
My application run some complex threads that fetch maps in a background thread and draw them. Sometimes if I run the app for a couple hours on a slow network I seem to be getting it into a weird state where all my threads status are showing TimedWait or Wait (except the ones that are Native such as main).
What is the cause of this? How can I debug it? I am absolutely lost and I know this is a bit of a general question but I would appreciate it if someone could point me to the right direction. EG:
How to pin point the cause of the problem.
What king of issues generally cause all the threads to lock up?
Anybody seen anything similar?
Thanks
A timed wait is simply a thread which is blocked on some O/S level call which has a timeout specified, such as a simple wait primitive (Object.wait()) or a socket operation (Socket read()/write()), a thread queue etc. It's quite normal for any complex program to have several or many of these - I have an application server which routinely has hundreds, even thousands.
Your threads may be backing up on non-responsive connections and may not be misbehaving at all, per se. It may simply be that you need to program them to detect and abort an idle connection.
Click on each of the threads which you are concerned about and analyze their stack trace for how they got there.
Most decent profiling tools (and application containers) will have the option of printing a full stack trace, and more modern ones will do a dead-lock and live-lock analysis for you. The JVisualVM tool distributed with Sun's JDK and available on the net as VisualVM will do this and it's very effective. Most decent profilers will also show lock acquisition in the stack trace (yours, above, is not in that view).
Otherwise, you are looking for two or more threads contending for the same lock or acquiring the same locks in a different order. You may need to do this manually by actually examining the source and annotating your stack trace, but you should be able to whittle down likely candidates if your tool doesn't point right to the conflicting threads.
I am using samurai tool to analyze thread dump. It looks like it has many blocked threads. I have no clue to derive anything from the thread dump.
I have an SQL query in my Java application that runs on weblogic that takes enormous time to complete. After running this query by clicking on my Java application button several times hangs my JVM.
Thread dumps can be found # : http://www.megafileupload.com/en/file/379103/biserver2-txt.html
Can you help me understand what does the thread dump say ?
The amount of data you provide is a bit overwhelming, so let's just give you a hint how to proceed. For the analysis I use open source threadlogic application based on TDA. It takes few seconds to parse 3 MiB worth of data but in nicely shows 22 different stack trace dumps in one file:
Drilling down to reveals really disturbing list of warnings and alerts.
I don't have time to examine all of them, but here is a list of those marked as FATAL (keep in mind that false-positives are also to be expected):
Wait for SLSB Beans
Description: Waiting for Stateless Session Bean (SLSB) instance from the SLSB Free pool
Advice: Beans all in use, free pool size size insufficient
DEADLOCK
Description: Circular Lock Dependency Detected leading to Deadlock
Advice: Deadlock detected with circular dependency in locks, blocked threads will not recover without Server Restart. Fix the order of locking and or try to avoid locks or change order of locking at code level, Report with SR for Server/Product Code
Finalizer Thread Blocked
Description: Finalizer Thread Blocked
Advice: Check if the Finalizer Thread is blocked for a lock which can lead to wasted memory waiting to be reclaimed from Finalizer Queue
WLS Unicast Clustering unhealthy
Description: Unicast messaging among Cluster members is not healthy
Advice: Unicast group members are unable to communicate properly, apply latest Unicast related patches and enable Message Ordering or switch to Multicast
WLS Muxer is processing server requests
Description: WLS Muxer is handling subsystem requests
Advice: WLS Server health is unhealthy as some subsystems are overwhelmed with requests which is leading to the Muxer threads directly handling requests. instead of dispatching to relevant subsystems. There is likely a bug here.
Stuck Thread
Description: Thread is Stuck, request taking very long time to finish
Advice: Check why the thread or call is taking very long??. Is it blocked for unavailable or bad resource or contending for Lock?. Can be ignored if it is doing repeat work in a loop. (like adapter threads polling for events in a infinite loop)...
The issue was with WLDF logging information to log file. Once disabled it helped improve performance enormously. I am not a fan of ThreadLogic as a tool for thread dump analysis. It shows circular deadlock when you have stuck threads no matter how variant the issue is.
Thread dumps are the snapshot of all threads running in the application at given moment. Thread dump will have hundreds/thousands of application threads. It would be hard to scroll through every single line of the stack trace in every single thread. Call Stack Tree consolidates all the threads stack trace into one single tree and gives you one single view. It makes the thread dumps navigation much simpler and easier. Below is the sample call stack tree generated by fastThread.io.
Fig 1: Call stack Tree
You can keep drilling down to see code execution path. Fig 2 shows the drilled down version of a particular branch in the Call Stack Tree diagram.
Fig 2: Drilled down Call Stack Tree
Sample call stack tree generated by FastThread.io
I am analyzing the differences between approaches for taking thread dumps. Below are the couple of them I am researching on
Defining a jmx bean which triggers jstack through Runtime.exec() on clicking a declared bean operation.
Daemon thread executing "ManagementFactory.getThreadMXBean().dumpAllThreads(true, true)" repeatedly after a predefined interval.
Comparing the thread dump outputs between the two, I see the below disadvantages with approach 2
Thread dumps logged with approach 2 cannot be parsed by open source thread dump analyzers like TDA
The ouput does not include the native thread id which could be useful in analyzing high cpu issues (right?)
Any more?
I would appreciate to get suggestions/inputs on
Are there any disadvantages of executing jstack through Runtime.exec() in production code? any compatibility issues on various operating systems - windows, linux?
Any other approach to take thread dumps?
Thank you.
Edit -
A combined approach of 1 and 2 seems to be the way to go. We can have a dedicated thread running in background and printing the thread dumps in the log file in a format understood by the thread dump analyzers.
If any extra information is need (like say probably the native thread id) which is logged only by the jstack output, we do it manually as required.
You can use
jstack {pid} > stack-trace.log
running as the user on the box where the process is running.
If you run this multiple times you can use a diff to see which threads are active more easily.
For analysing the stack traces I use the following sampled periodically in a dedicated thread.
Map<Thread, StackTraceElement[]> allStackTraces = Thread.getAllStackTraces();
Using this information you can obtain the thread's id, run state and compare the stack traces.
With Java 8 in picture, jcmd is the preferred approach.
jcmd <PID> Thread.print
Following is the snippet from Oracle documentation :
The release of JDK 8 introduced Java Mission Control, Java Flight Recorder, and jcmd utility for diagnosing problems with JVM and Java applications. It is suggested to use the latest utility, jcmd instead of the previous jstack utility for enhanced diagnostics and reduced performance overhead.
However, shipping this with the application may be licensing implications which I am not sure.
If its a *nix I'd try kill -3 <PID>, but then you need to know the process id and maybe you don't have access to console?
I'd suggest you do all the heap analysis on a staging environment if there is such an env, then reflect your required Application Server tuning on production if any. If you need the dumps for analysis of your application's memory utilization, then perhaps you should consider profiling it for a better analysis.
Heap dumps are usually generated as a result of OutOfMemoryExceptions resulting from memory leaks and bad memory management.
Check your Application Server's documentation, most modern servers have means for producing dumps at runtime aside from the normal cause I mentioned earlier, the resulting dump might be vendor specific though.
I have a multithreaded application. Several messages are coming to the application and are processed in separated threads. For this I am using classes ThreadPoolExecutor and FutureTask from package java.util.concurrent.
Occasionally I have some deadlocks in the application. When a deadlock occurs I want to interrupt the blocking thread and I want to log the stack trace of this thread so that I can later resolve the deadlock.
Is there any way how can we find the stack trace of a thread outside of that thread in Java?
See here for how to generate stack traces, including how to do this programatically. From the console, Ctrl+Break will dump the stack traces to stdout. See also this SO question for more details.
You could log the stack traces of all thread from time to time (or before killing the process) from within your application. To do that use:
Map<Thread, StackTraceElement[]> m = Thread.getAllStackTraces();
for(Map.Entry<Thread, StackTraceElement[]> e : m.entrySet()) {
log(e.getKey().toString());
for (StackTraceElement s : e.getValue()) {
log(" " + s);
}
}
When running nightly automated tests, sometimes some one of the test cases gets into a deadlock. I added a "TimeBomb" daemon thread that waits 30 minutes, and if then logs all stack traces as above.
Before entering the deadlock region, set a field like,
thread = Thread.currentThread();
In your monitoring thread you can perform thread.getStackTrace(); to get the stack trace of that thread at any time.
You use JStack. Here is a nice blog entry that details how to get stack traces.
I wasn't sure if you wish to obtain the stacktrace from within the same JVM or externally, but if you wish to obtain the stack trace with external tools, the following will help:
The Java VisualVM tool can be used to connect to the running JVM, where the thread stack can be dumped. This is usually the preferred approach for most people using Java 6. Once VisualVM is launched, the thread dump of the process can be obtained by selecting the process in the Application tab. A Threads tab is now made available to view the threads in the running process, and in the same tab you'll find the "Thread Dump" button to extract the required information.
The jstack utility within the JDK can also be used to produce thread stacktraces.
When a deadlock occurs I want to interrupt the blocking thread ...
You could implement a periodic task to check for deadlocks (where deadlocks are java intrinsic or Lock based) and call interrupt on all threads involved in the scenario. However, this has no guarantees that it will solve your problem. Its likely the scenario that will just happen again. See Dr Heinz's article on a deadlock detector for details.
If fact, there is no guarantee that interrupt will even free up a blocked process like this. Its a far better approach to avoid the deadlock scenario in the first place by, for example, using locks with timeouts and retry strategies or 'try before you buy' approaches.
and I want to log the stack trace of this thread...
If you want to do this programatically, again, follow Dr Heinz's example. If not, just generate the thread dump when you've spotted the problem.
Is there any way how can we find the stack trace of a thread outside of that thread in Java?
Yes and no. You can dump the threads from other VMs but their stack traces may not be as useful as you might think to determining the causes of your deadlock. If a genuine deadlock has been detected (by the JVM itself on thread dump of your applications VM) you should have everything you need to debug the cause (more or less).