This question already has answers here:
ExecutorService that interrupts tasks after a timeout
(11 answers)
Closed 7 years ago.
This is just an example to explain my problem...
I am using ExecutorService with 20 active threads and 75K max queued items...
In my case, a normal task should not take more than 10 seconds, if it takes more time that means there's some problem with the task.
If all the threads are hung due to problematic tasks my RejectionHandler would restart the entire service.
I have two questions here:
I do not like the idea of restarting the service, instead if there's
way to detect hanging thread and we could just restart that hung
thread that would be great. I have gone through couple of articles to handle hung threads with ThreadManager but have not found anything
with ExecutorService.
I am very much fascinated about the Executors.newCachedThredPool()
because on peak days we are heavily loaded with incoming tasks, and
on other days they are very few. Any suggestions would be greatly
appreciated.
public class HangingThreadTest {
// ExecutorService executorService = Executors.newCachedThreadPool()
private static ExecutorService executorService = new ThreadPoolExecutor(10,
20, 5L, TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(75000));
public static void main(String... arg0) {
for (int i = 0; i < 50000; i++) {
executorService.submit(new Task());
}
}
}
/**
* Task to be completed
*/
class Task implements Runnable {
private static int count = 0;
#Override
public void run() {
count++;
if (count%5 == 0) {
try {
System.out.println("Hanging Thread task that needs to be reprocessed: "
+ Thread.currentThread().getName()+" count: "+count);
Thread.sleep(11000);
} catch (InterruptedException e) {
// Do something
}
}
else{
System.out.println("Normal Thread: "
+ Thread.currentThread().getName()+" count: "+count);
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
//Do something
}
}
}
}
There is no build-in mechanism in Executors framework that would help terminate a thread if it has been running for more than a threshold value.
But we can achieve this with some extra code as below:
Get the Future object returned by the executorService.submit(...);.
Future future = executorService.submit(new Task());
Call the get method on this future object to and make it wait only for threshold interval for task completion. Below, an example that is waits for only 2 secs.
try {
f.get(2, TimeUnit.SECONDS);
} catch (TimeoutException e) {
f.cancel(true);
} catch (Exception e) {}
The above code waits for 2 seconds for task completion it throws a TimeoutException if it doesn't get completed during that time. Subsequently we can call cancel method on the future object. This results in setting the interrupt flag in the thread that is executing the task.
Now the final change is, in the Task class code we need to check at necessary points (application dependent), whether the interrupt flag has been set to true using isInterrupted() method of Thread class. If interrupted==true, we can do the necessary clean up and return from the run method immediately. The critical piece here is to identify the necessary points in your Task class where you want to check for this interrupted flag.
This makes the thread available for processing next task.
You may have a look at this article, it was very helpful for me before when I was facing the same problem : Java Hanging Thread Detection
Related
I am attempting to reimplement my concurrent code using CyclicBarrier which is new to me. I can do without it but am time trialling it against my other solution, the problem I have is a deadlock situation with the following code:
//instance variables (fully initialised elsewhere).
private final ExecutorService exec = Executors.newFixedThreadPool(4);
private ArrayList<IListener> listeners = new ArrayList<IListener>();
private int[] playerIds;
private class WorldUpdater {
final CyclicBarrier barrier1;
final CyclicBarrier barrier2;
volatile boolean anyChange;
List<Callable<Void>> calls = new ArrayList<Callable<Void>>();
class SyncedCallable implements Callable<Void> {
final IListener listener;
private SyncedCallable(IListener listener) {
this.listener = listener;
}
#Override
public Void call() throws Exception {
listener.startUpdate();
if (barrier1.await() == 0) {
anyChange = processCommons();
}
barrier2.await();
listener.endUpdate(anyChange);
return null;
}
}
public WorldUpdater(ArrayList<IListener> listeners, int[] playerIds) {
barrier2 = new CyclicBarrier(listeners.size());
barrier1 = new CyclicBarrier(listeners.size());
for (int i : playerIds)
calls.add(new SyncedCallable(listeners.get(i)));
}
void start(){
try {
exec.invokeAll(calls);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
void someMethodCalledEveryFrame() {
//Calls some Fisher-something method that shuffles int[]
shufflePIDs();
WorldUpdater updater = new WorldUpdater(listeners, playerIds);
updater.start();
}
I use the debugger in Android Studio (intelliJ) to pause execution at this stage. I get multiple threads showing the my await calls as the last of my code to be executed
->Unsafe.park
->LockSupport.park
->AbstractQueuedSynchronizer$ConditionObject.await
->CyclicBarrier.doWait
->CyclicBarrier.await
At least one thread will be have this stack:
->Unsafe.park.
->LockSupport.park
->AbstractQueuedSynchronizer$ConditionObject.await
->LinkedBlockingQueue.take
->ThreadPoolExecutor.getTask
->ThreadPoolExecutor.runWorker
->ThreadPoolExecutor$Worker.run
->Thread.run
I notice that the CyclicBarrier plays no part in these latter stray threads.
processCommons is calling exec.invokeAll (on the 3 listeners), I suppose this means I am running out of threads. But many times this doesn't happen so please could someone clarify why ExecutorService cannot consistently schedule my threads? They have their own stack and program counter so I would have thought this to not be a problem. I only ever have max 4 running at once. Someone help me with the math?
What is the value of listeners.size() when your WorldUpdater is created? If it is more than four, then your threads will never get past the barrier.
Your ExecutorService has exactly four threads. No more, no fewer. The callers of barrier1.await() and barrier2.await() will not get past the barrier until exactly listeners.size() threads are waiting.
My gut reaction is, it would be a mistake for pool threads to use a CyclicBarrier. CyclicBarrier is only useful when you know exactly how many threads will be using it. But, when you're using a thread pool, you often do not know the size of the pool. In fact, in a real-world (i.e., commercial) application, if you're using a thread pool, It probably was not created by your code at all. It probably was created somewhere else, and passed in to your code as an injected dependency.
I did a little experiment and came up with:
#Override
public Void call() throws Exception {
System.out.println("startUpdate, Thread:" + Thread.currentThread());
listener.startUpdate();
if (barrier1.await() == 0) {
System.out.println("processCommons, Thread:" + Thread.currentThread());
anyChange = processCommons();
}
barrier2.await();
System.out.println("endUpdate, Thread:" + Thread.currentThread());
listener.endUpdate(anyChange);
return null;
}
Which revealed when using a pool of 3 with 3 listeners, I will always hang in processCommons which contains the following:
List<Callable<Void>> calls = new ArrayList<Callable<Void>>();
for (IListener listiner : listeners)
calls.add(new CommonsCallable(listener));
try {
exec.invokeAll(calls);
} catch (InterruptedException e) {
e.printStackTrace();
}
With 2 threads waiting at the barrier and the third attempting to create 3 more. I needed one extra thread in the ExecutorService and the 2 at the barrier could be "recycled" as I was asking in my question. I've got references to 6 threads at this stage when exec is only holding 4. This can run happily for many minutes.
private final ExecutorService exec = Executors.newFixedThreadPool(8);
Should be better, but it was not.
Finally I did breakpoint stepping in intelliJ (thanks ideaC!)
The problem is
if (barrier1.await() == 0) {
anyChange = processCommons();
}
barrier2.await();
Between the 2 await you may get several suspended threads that haven't actually reached the await. In the case of 3 listeners out of a pool of 4 it only takes one to get "unscheduled" (or whatever) and barrier2 will never get the full complement. But what about when I have a pool of 8? The same behaviour manifests with all but two of the threads the stack of limbo:
->Unsafe.park.
->LockSupport.park
->AbstractQueuedSynchronizer$ConditionObject.await
->LinkedBlockingQueue.take
->ThreadPoolExecutor.getTask
->ThreadPoolExecutor.runWorker
->ThreadPoolExecutor$Worker.run
->Thread.run
What can be happening here to disable all 5 threads? I should have taken James Large's advice and avoided crowbarring in this over elaborate CyclicBarrier.--UPDATE-- It can run all night now without CyclicBarrier.
I'm using a global Executor service with some fixed thread pool size. We have bunch of related tasks that we submit for execution and wait on list of futures.
Recently, we faced a high CPU utilization issue and on debugging I found that an exception occurred while calling get() on one of the item in list of futures. Current, we iterate over the list and there is a try catch surrounding the whole loop.
try{
List<Result> results = new ArrayList<>()
for(Future<Result> futureResult: futureResults{
Result result = futureResult.get();
results.add(result);
}
} catch(Exception e){
throw new InternalServiceException(e);
}
//Do something with results
Wanted to know the behaviour of other threads if get is never called on some of the items in future. I tried searching but was not able to find anything.
Also, can this behaviour trigger high CPU utilization ?
http://www.journaldev.com/1650/java-futuretask-example-program
I would still check if the future isDone as in the example above.
If you need to run other operations or want to utilize the CPU better then I would put the collector in a separate thread and perhaps just poll for results every minute or so.
Could be scheduled or handled by Thread.sleep.
Executors class provides various methods to execute Callable in a thread pool. Since callable tasks run in parallel, we have to wait for the returned Object.
Callable tasks return java.util.concurrent.Future object. Using Future we can find out the status of the Callable task and get the returned Object.
It provides get() method that can wait for the Callable to finish and then return the result.
There is an overloaded version of get() method where we can specify the time to wait for the result, it’s useful to avoid current thread getting blocked for longer time.
Future provides cancel() method to cancel the associated Callable task. There are isDone() and isCancelled() methods to find out the current status of associated Callable task.
Here is a simple example of Callable task that returns the name of thread executing the task after some random time.
We are using Executor framework to execute 10 tasks in parallel and use Future to get the result of the submitted tasks.
public class FutureObjectTest implements Callable<String>{
#Override
public String call() throws Exception {
long waitTime = (long) (Math.random()*10000);
System.out.println(Thread.currentThread().getName() + " waiting time in MILISECONDS " + waitTime);
Thread.sleep(waitTime);
return Thread.currentThread().getName() + " exiting call method.";
}
public static void main(String [] args){
List<Future<String>> futureObjectList = new ArrayList<Future<String>>();
ExecutorService executorService = Executors.newFixedThreadPool(5);
Callable<String> futureObjectTest = new FutureObjectTest();
for(int i=0; i<10; i++){
Future<String> futureResult = executorService.submit(futureObjectTest);
futureObjectList.add(futureResult);
}
for(Future<String> futureObj : futureObjectList){
try {
System.out.println(futureObj.get());
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
}
System.out.println("Starting get method of wait");
////////////get(Timeout) method///////
futureObjectList.clear();
for(int i=0; i<10; i++){
Future<String> futureResult = executorService.submit(futureObjectTest);
futureObjectList.add(futureResult);
}
executorService.shutdown();
for(Future<String> futureObj : futureObjectList){
try {
System.out.println(futureObj.get(2000,TimeUnit.MILLISECONDS));
} catch (InterruptedException | ExecutionException | TimeoutException e) {
e.printStackTrace();
}
}
}
}
Is there a built-in way to cancel a Runnable task that has been scheduled at a fixed rate via ScheduledExecutorService.scheduleAtFixedRate and await it's completion if it happens to be running when cancel is called?.
Consider the following example:
public static void main(String[] args) throws InterruptedException, ExecutionException {
Runnable fiveSecondTask = new Runnable() {
#Override
public void run() {
System.out.println("5 second task started");
long finishTime = System.currentTimeMillis() + 5_000;
while (System.currentTimeMillis() < finishTime);
System.out.println("5 second task finished");
}
};
ScheduledExecutorService exec = Executors.newSingleThreadScheduledExecutor();
ScheduledFuture<?> fut = exec.scheduleAtFixedRate(fiveSecondTask, 0, 1, TimeUnit.SECONDS);
Thread.sleep(1_000);
System.out.print("Cancelling task..");
fut.cancel(true);
System.out.println("done");
System.out.println("isCancelled : " + fut.isCancelled());
System.out.println("isDone : " + fut.isDone());
try {
fut.get();
System.out.println("get : didn't throw exception");
}
catch (CancellationException e) {
System.out.println("get : threw exception");
}
}
The output of this program is:
5 second task started
Cancelling task..done
isCancelled : true
isDone : true
get : threw exception
5 second task finished
Setting a shared volatile flag seems the simplest option, but I'd prefer to avoid it if possible.
Does the java.util.concurrent framework have this capability built in?
I am not entirely sure what are you trying to achieve but as I went here from google search I thought It may be worth responding to your question.
1) If you want to forcibly stop heavy workload - unfortunately it seems there is no solution for it(when thread does not respond to interrupts). Only way of dealing with it would be to insert Thread.sleep(1) in between time consuming operations in your loop (http://docs.oracle.com/javase/1.5.0/docs/guide/misc/threadPrimitiveDeprecation.html) - maybe deamon thread would help here but I really discourage using them.
2) If you want to block current thread until the child thread finishes then instead of calling cancel you can use get http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html#get() or even get with timeout.
3) If you want clean cancel of subthread then you can call:
fut.cancel(false);
this will not interrupt current execution but will not schedule it to run again.
4) If your workload is not heavy and you only need to wait for 5 seconds then use thread sleep or TimeUnit sleep. In such case interrupt / cancel will be immediate.
Also your example lacking shutdown call on Executor which cause application does not stop.
I'm doing an optimization problem in which I want to execute each Solver thread (one at a time) with random parameters for a fixed period of time. If any of the thread successfully finds a solution, it would return and the program will exit.
I have the code below where I used an ExecutorService and Future to help me accomplish this. However, for some reason, the memory usage of the program increases linearly as time goes on, and the program will terminate with an OutOfMemory error before it gets very far. My Solver code is certainly not the issue as it has no static variables and uses a constant amount of memory. I'm wondering if it's because I'm not cleaning up the threads or handling the exceptions properly, but I can't seem to find any egregious problem from the code.
public class RandomizedSolver {
public static void main(String[] args) {
try {
for (int i = 0; i < 300; i++) {
ExecutorService executor = Executors.newSingleThreadExecutor();
try {
System.out.println("Starting new thread");
Future<Void> future = executor.submit(new Solver(args));
future.get(1, TimeUnit.SECONDS);
executor.shutdownNow();
break;
} catch (TimeoutException e) {
System.out.println("Thread timeout.");
executor.shutdownNow();
continue;
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
The point of using ExecutorServices is to reuse their threads, not to keep recreating them. You should revise your design and have only one ExecutorService, with the appropriate number of underlying threads, and submit all you tasks to that unique ExecutorService.
Also note that if your tasks take more than 1 seconds and if they do not terminate promptly when interrupted, you could have up to 300 ExecutorServices and 300 Solver tasks running at the same time. Depending on how much memory you Solver takes, that could result in a OOME.
I need to create async thread that runs once with a delay of 2 minutes and that can be killed at any moment. I saw several possible solutions:
ScheduledExecutorService and FutureTask allow me to interrupt a running task, but I will have to invoke shutdown() to terminate all the running threads, and this will block user until the processes were terminated. Also, I will have to frequently invoke Thread.interrupted() as described in Enno Shioji's answer.
Timer and TimerTask do not require to release running threads, but I have no way to interrupt a running timer thread (Timer.cancel() just cancels future scheduling)
Using Thread and sleep with thread interruption problem.
Is there a good solution? (I'm using tomcat 7)
thank you
After some tests and researches, FutureTask.cancel() and Threads need similar handling of interrupts, as stated in Enno Shioji's answer
Check interruption flag in your logic
Act upon Interrupted exception
An example that tests interruption flag:
private final class MyTask implements Runnable {
public void run() {
try{
for(int j=0; j<100000000; j++) {
for(int i=1; i<1000000000; i++){
if(Thread.interrupted()){ //Don't use Thread.interrupt()!
Log.debug("Thread was interrupted for" + cache);
return; //Stop doing what you are doing and terminate.
}
Math.asin(0.1565365897770/i);
Math.tan(0.4567894289/i);
}
}
}catch(Throwable e){//if exception is uncaught, the scheduler may not run again
...
}
}
}
As I understand, ScheduledExecutorService maybe be shutdown when application ends running
For your scenario 2 with TimerTask, why not just return from the run() method after calling this.cancel()?
Here's a snippet from something I wrote. I use the same technique whenever the tool encounters a situation that would make further execution invalid, like a misconfiguration.
...
if ( count < 1 ) {
logger.error("CANCELING THREAD FOR " + host + ":" + jmxPort + "! " +
"- CONFIGURATION INCOMPLETE - DUMP_COUNT must be 1 or greater.");
this.cancel();
return;
}
...
Using Option 1 you can use FutureTask.cancel() with mayInterruptIfRunning parameter set to true to cancel your tasks.
The ScheduledExecutorService.scheduleAtFixedRate() creates a ScheduledFuture, which still can be canceled trough the FutureTask api.
Here is a naive test I've used to verify that task gets canceled. I suppose the worker thread may not be interrupted if you do some blocking IO (network or disk), but I haven't tested it. If cancel is called while task is not running, it all stops nicely, but if task is running when cancel is called the executor will try to kill the thread.
public static void main(String[] args) throws InterruptedException {
ScheduledThreadPoolExecutor executor = new ScheduledThreadPoolExecutor(2);
ScheduledFuture<?> future = executor.scheduleAtFixedRate(new Runnable() {
int i = 0;
public void run() {
int j = i++;
System.err.println("Run " + j);
try {
Thread.sleep(5000L);
} catch (InterruptedException e) {
System.err.println("Interrupted " + j);
}
}
}, 1000L, 2000L, TimeUnit.MILLISECONDS);
Thread.sleep(10000L);
System.err.println("Canceled " + future.cancel(true));
Thread.sleep(20000L);
executor.shutdownNow();
System.err.println("Finished");
}