I am using some 3rd party code, which may hang indefinitely in some cases. This leads to hanging threads which keep holding the resources and clogging a thread pool.
Eventually thread pool becomes full of useless threads and system effectively fails.
In Java one can't forcefully kill the thread (kill -9). But how then to manage such edge cases?
Obviously fixing the bug would be better, however alternative include
only run the 3rd party code/library in a sub-process. Just killing the thread is unlikely to be enough.
you could hack the 3rd party code to check for interrupts in the sections you find run for too long. You can take a stack trace to run out where this is.
Use Thread.stop() though this has been disabled in Java 8.
when you detect there is a hung thread, increase the size of the thread pool by one. This will give you the correct number of active threads.
Related
I've a java process running and unfortunately one thread inside the process is hung.
I found the Thread id which was hung using jstack, however I was unable to find any references on how to interrupt this thread using the id?
Is it possible to Interrupt/Stop a thread from console (or basically outside the process) by using the processId and ThreadId?
Any suggestions on how to tackle this?
PS : I don't want to kill the process as its just one thread which is hung. Also, neither do I want to make code changes to Stop/Interrupt the thread. I just want to kill it, so all its resources can be released.
There's no baked-in way to kill a thread within the JVM, at least not a deliberately implemented one.
Having said that, if you have started your JVM with the appropriate parameters, so that you can start a remote JMX session to it, you can actually suspend the thread and inject a RuntimeException into it, which will almost surely terminate it (unless you are doing something gnarly with RuntimeExceptions in it).
See this blogpost.
P.S. You would never start your JVM in production allowing rogue JMX connections though, and if you're not in production, I'd guess that the above approach is not of much help to you.
My application run some complex threads that fetch maps in a background thread and draw them. Sometimes if I run the app for a couple hours on a slow network I seem to be getting it into a weird state where all my threads status are showing TimedWait or Wait (except the ones that are Native such as main).
What is the cause of this? How can I debug it? I am absolutely lost and I know this is a bit of a general question but I would appreciate it if someone could point me to the right direction. EG:
How to pin point the cause of the problem.
What king of issues generally cause all the threads to lock up?
Anybody seen anything similar?
Thanks
A timed wait is simply a thread which is blocked on some O/S level call which has a timeout specified, such as a simple wait primitive (Object.wait()) or a socket operation (Socket read()/write()), a thread queue etc. It's quite normal for any complex program to have several or many of these - I have an application server which routinely has hundreds, even thousands.
Your threads may be backing up on non-responsive connections and may not be misbehaving at all, per se. It may simply be that you need to program them to detect and abort an idle connection.
Click on each of the threads which you are concerned about and analyze their stack trace for how they got there.
Most decent profiling tools (and application containers) will have the option of printing a full stack trace, and more modern ones will do a dead-lock and live-lock analysis for you. The JVisualVM tool distributed with Sun's JDK and available on the net as VisualVM will do this and it's very effective. Most decent profilers will also show lock acquisition in the stack trace (yours, above, is not in that view).
Otherwise, you are looking for two or more threads contending for the same lock or acquiring the same locks in a different order. You may need to do this manually by actually examining the source and annotating your stack trace, but you should be able to whittle down likely candidates if your tool doesn't point right to the conflicting threads.
I have a simple test run of some medium-complexity code that won't terminate, i.e. the main method finishes but the process does not die.
Here is a rundown of the code (which is too long to be pasted here):
ProcessBuilder is used to create a bunch of subprocesses. They all die properly (if you can believe VisualVM).
We use log4j.
The main algorithm runs inside a FutureTask on which run and later get are called.
We don't explicitly use RMI, even though the list of threads seems to suggest so.
Obviously, I can call System.exit(0), but I'd like to know what is amiss here. I have not been able to produce a minimum failing example. Also, I can not identify an obvious culprit from the thread list; maybe you can?
Edit: See here for a thread dump.
Scorpion lead me to the right answer:
RMI Reaper is something like a garbage collector for remote objects, e.g. instances of (subclasses of) UnicastRemoteObject. It is a non-daemon thread and therefore blocks JVM termination if there are still exported objects which can not be cleaned up.
You can explicity force remote objects to be cleaned up in this sense by passing them to UnicastRemoteObject.unexportObject(., true). If you do this on all previously exported objects, RMI Reaper terminates and JVM is free to shut down.
You mention FutureTask. The first thing that comes to my mind is: are you using ExecutorService and forgetting to shut it down?
The second thing that comes to my mind is: are you reading to the end all the streams from the process? I worked with subprocesses long ago, and I don't remember exactly, but. I had problems similar to what you described, and by reading the streams to the end the problem would misteriously disappear!
Exposition:
I think the Java VM is awesome. It's guarantee of the safety of bytecode, the the standard libraries, ... are amazing, especially the ability to load a Java class on the fly, and know that it can't crash the VM (good luck with *.so files or kernel modules).
One thing I don't understand, is how Java treats Thread.stop
I've read http://java.sun.com/j2se/1.5.0/docs/guide/misc/threadPrimitiveDeprecation.html
but it seems weird for the following reasons:
1) Resource Management
On Unix OS, if a process is hogging up resources, I can kill -9 it.
2) Breaking of Abstraction:
If I start a computationally expensive job, and I no longer need the computation, I can kill -9 it. Under this Java threading model, my computation thread has to periodically check for some boolean flag to see whether it should quit [this seems like breaking abstraction layers -- when I'm writing computation code, I should focus on computation code, not where to spread out checks for whether it should terminate.
3) Safety of Lock/Monitors
So the official reason is "what is a thread is holding a Lock/Monitor and it gets Thread.stopped ? The objects will be left in damaged states" -- yet, in OSes this is not a problem, we have interrupt handlers. Why can't Java threads have interrupt handlers that work like OS interrupt handlers?
Question:
Clearly, I am thinking about Java Threads with the wrong mental model. How should I be thinking about Java threads?
Thanks!
I think the key thing to remember is that threads are not processes. The only thing that threads "own" is the execution thread, since everything else can potentially be shared with other threads within the same process (memory space). Stopping a thread can't clean anything up because things may still be completely valid for other threads to handle.
In an operating system process, the OS keeps track of everything that the process owns (memory, files, locks, etc) and cleans things up properly when you SIGKILL a process.
You seems to be confusing threads and processes.
Resources(Allocated memory, Locks, file handles and so on) belong to the process not to any specific thread in the process. The entire point(And danger) of threads is that multiple threads within the same process share resources.
So it does not sense to talk about any resource belonging to a single thread.
ps: I am pretty sure you can't use kill/kill -9 to kill a thread in Linux. The man page for kill say it can only kill a process or process groups.
Is it possible to kill a Java thread without raising an exception in it?
This is just for testing purposes - I want to simulate a case when the entire computer dies mid-thread.
Note - I saw a deprecated Thread.destroy() method, but the documentation says it never got implemented in the first place.
No. There is the deprecated, 'inherently unsafe' Thread.stop() method, but as its comment emphasizes, things could be left in an deeply corrupted state, and the ThreadDeath Error is still thrown inside the thread.
Sun's explanation of the problems with stop(), which can manifest long after it appears to work, is at:
http://java.sun.com/j2se/1.5.0/docs/guide/misc/threadPrimitiveDeprecation.html
Wouldn't killing the JVM process (or yanking the plug) be a better simulation of computer death?
There is no portable method. You might try to call "kill -9" (or your local equivalent) on the whole java process, if you want to suppress the running of finalizers and shutdown hooks.
You won't get any kind of repeatable results out of such a test, but it might be interesting to perform such tests a few thousand times if your program is writing to the file system or a database and might leave inconsistent data structures when being killed.
Or you could... kill the process. (ie, if this is Linux, send a kill -9 signal to the process).
Beware the race issues if you're trying to test something - if you hoping to crash badly - it might only do it once a month if you're particularly unlucky.
What is the point of your test? Java offers no guarantees about what happens on an exit, apart from attempting to run shutdown hooks if the exit is a "clean" one
It sounds akin to trying to test your program's behaviour in the case it goes OutOfMemory; there's nothing you can do about it and no way of telling deterministically what will happen
Is there any reason you can't use Thread.suspend()? It will stop the thread so you can examine the state when the thread is interrupted.
You could also use Thread.stop() although that risks throwing multiple ThreadDeathExceptions. You can wrap it int try/catch/finally blocks, but there are no guarantees.
Thread stop() throws an error in the thread. ThreadDeath
The only way to simulate an application dying mid thread is to call System.exit().
However, this is pretty random so you have to perform the test many times to have any confidence you application behaves correctly no matter where it dies.