The main process:
int cpus = Runtime.getRuntime().availableProcessors();
ExecutorService executor = Executors.newFixedThreadPool(cpus);
List<Callable<Object>> todo = new ArrayList<Callable<Object>>(lines.size());
for (int r = 0; r < lines.size(); r++) {
String filename = r + 1 + "";
todo.add(Executors.callable(new AppConsole(filename, lines.get(r))));
}
List<Future<Object>> answers = executor.invokeAll(todo);
The AppConsole class implements Runnable and the overriden run method is as follows:
public void run() {
try{
} catch (SecurityException exception) {
exception.printStackTrace();
} catch (FileNotFoundException exception) {
exception.printStackTrace();
} catch (IOException exception) {
exception.printStackTrace();
} catch (SAXException exception) {
exception.printStackTrace();
} catch (Exception exception) {
exception.printStackTrace();
} finally {
if (output != null) {
output.close();
}
}
}
}
The main process is suspended and cannot finish because of a connection timeout exception thrown from one of the threads. Now I can see the cpu usage drops to 0% and the memory consumption keeps at the same level when it got suspended.
Could someone please help me solve this problem? At least, help me make the main process finish?
Throwing an exception frees up the tasks and the main thread. The ExecutorService treats an Exception throw much like another return value and doesn't have a problem handling it.
The main thread will only block waiting for one of your tasks to complete. I would look at the tasks/threads which are still active to see what they are doing e.g. perhaps they haven't timed out yet.
you could use the alternative version of invokeAll that takes an explicit timeout value :
http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/ExecutorService.html
invokeAll(Collection<? extends Callable<T>> tasks,
long timeout,
TimeUnit unit)
this way you can force all of your tasks to timeout after, say, 30 seconds. Then you can inspect the Futures returned to see which have completed.
You just need to add:
executor.shutdown();
This waits when all tasks will be completed and shutdown all threads, application will exit after that.
Please see javadoc for newFixedThreadPool:
Creates a thread pool that reuses a fixed number of threads operating
off a shared unbounded queue. At any point, at most nThreads threads
will be active processing tasks. If additional tasks are submitted
when all threads are active, they will wait in the queue until a
thread is available. If any thread terminates due to a failure during
execution prior to shutdown, a new one will take its place if needed
to execute subsequent tasks. The threads in the pool will exist until
it is explicitly shutdown.
Related
Our Java application is having an issue where it blocks indefinitely when it tries to write to a log file located on a NFS share and the NFS share is down.
I was wondering whether we could solve this problem by having a Future execute the write operation with a timeout. Here is a little test program I wrote:
public class write_with_future {
public static void main(String[] args) {
int iteration=0;
while (true) {
System.out.println("iteration " + ++iteration);
ExecutorService executorService = Executors.newSingleThreadExecutor();
Future future = executorService.submit(new Runnable() {
public void run() {
try {
Category fileLogCategory = Category.getInstance("name");
FileAppender fileAppender = new FileAppender(new SimpleLayout(), "/usr/local/app/log/write_with_future.log");
fileLogCategory.addAppender(fileAppender);
fileLogCategory.log(Priority.INFO, System.currentTimeMillis());
fileLogCategory.removeAppender(fileAppender);
fileAppender.close();
}
catch (IOException e) {
System.out.println("IOException: " + e);
}
}
});
try {
future.get(100L, TimeUnit.MILLISECONDS);
}
catch (InterruptedException ie) {
System.out.println("Current thread interrupted while waiting for task to complete: " + ie);
}
catch (ExecutionException ee) {
System.out.println("Exception from task: " + ee);
}
catch (TimeoutException te) {
System.out.println("Task timed out: " + te);
}
finally {
future.cancel(true);
}
executorService.shutdownNow();
}
}
}
When I ran this program with a maximum heap size of 1 MB, and the NFS share was up, this program was able to execute over 1 million iterations before I stopped it.
But when I ran the program with a maximum heap size of 1 MB, and the NFS share was down, the program executed 584 iterations, getting a TimeoutException each time, and then it failed with a java.lang.OutOfMemoryError error. So I am thinking that even though future.cancel(true) and executorService.shutdownNow() are being called, the executor threads are blocked on the write and not responding to the interrupts, and the program eventually runs out of memory.
Is there any way to clean up the executor threads that are blocked?
If appears that Thread.interrupt() does not interrupt threads blocked in an I/O operation on an NFS file. You might want check the NFS mount options, but I suspect that you won't be able to fix that problem.
However, you could certainly prevent it from causing OOME's. The reason you are getting those is that you are not using ExecutorServices as they are designed to be used. What you are doing is repeatedly creating and shutting down single thread services. What you should be doing is creating on instance with a bounded thread pool and using that for all of the tasks. If you do it that way, if one of the threads takes a long time ... or is blocked in I/O ... you won't get a build-up of threads, and run out of memory. Instead, the backlogged tasks will sit in the ExecutorService's work queue until one of the worker thread unblocks.
I want to use the Executor interface (using Callable) in order to start a Thread (let's call it callable Thread) which will do work that uses blocking methods.
That means the callable Thread can throw an InterruptedException when the main Thread calls the Future.cancel(true) (which calls a Thread.interrupt()).
I also want my callable Thread to properly terminate when interrupted USING other blocking methods in a cancellation part of code.
While implementing this, I experienced the following behavior: When I call Future.cancel(true) method, the callable Thread is correctly notified of the interruption BUT if the main Thread immediately waits for its termination using Future.get(), the callable Thread is kind of killed when calling any blocking method.
The following JUnit 5 snippet illustrates the problem.
We can easily reproduce it if the main Thread does not sleep between the cancel() and the get() calls.
If we sleep a while but not enough, we can see the callable Thread doing half of its cancellation work.
If we sleep enough, the callable Thread properly completes its cancellation work.
Note 1: I checked the interrupted status of the callable Thread: it is correctly set once and only once, as expected.
Note 2: When debugging step by step my callable Thread after interruption (when passing into the cancellation code), I "loose" it after several step when entering a blocking method (no InterruptedException seems to be thrown).
#Test
public void testCallable() {
ExecutorService executorService = Executors.newSingleThreadExecutor();
System.out.println("Main thread: Submitting callable...");
final Future<Void> future = executorService.submit(() -> {
boolean interrupted = Thread.interrupted();
while (!interrupted) {
System.out.println("Callable thread: working...");
try {
Thread.sleep(500);
} catch (InterruptedException e) {
System.out.println("Callable thread: Interrupted while sleeping, starting cancellation...");
Thread.currentThread().interrupt();
}
interrupted = Thread.interrupted();
}
final int steps = 5;
for (int i=0; i<steps; ++i) {
System.out.println(String.format("Callable thread: Cancelling (step %d/%d)...", i+1, steps));
try {
Thread.sleep(200);
} catch (InterruptedException e) {
Assertions.fail("Callable thread: Should not be interrupted!");
}
}
return null;
});
final int mainThreadSleepBeforeCancelMs = 2000;
System.out.println(String.format("Main thread: Callable submitted, sleeping %d ms...", mainThreadSleepBeforeCancelMs));
try {
Thread.sleep(mainThreadSleepBeforeCancelMs);
} catch (InterruptedException e) {
Assertions.fail("Main thread: interrupted while sleeping.");
}
System.out.println("Main thread: Cancelling callable...");
future.cancel(true);
System.out.println("Main thread: Cancelable just cancelled.");
// Waiting "manually" helps to test error cases:
// - Setting to 0 (no wait) will prevent the callable thread to correctly terminate;
// - Setting to 500 will prevent the callable thread to correctly terminate (but some cancel process is done);
// - Setting to 1500 will let the callable thread to correctly terminate.
final int mainThreadSleepBeforeGetMs = 0;
try {
Thread.sleep(mainThreadSleepBeforeGetMs);
} catch (InterruptedException e) {
Assertions.fail("Main thread: interrupted while sleeping.");
}
System.out.println("Main thread: calling future.get()...");
try {
future.get();
} catch (InterruptedException e) {
System.out.println("Main thread: Future.get() interrupted: Error.");
} catch (ExecutionException e) {
System.out.println("Main thread: Future.get() threw an ExecutionException: Error.");
} catch (CancellationException e) {
System.out.println("Main thread: Future.get() threw an CancellationException: OK.");
}
executorService.shutdown();
}
When you call get() on a canceled Future, you will get a CancellationException, hence will not wait for the Callable’s code to perform its cleanup. Then, you are just returning and the observed behavior of threads being killed seems to be part of JUnit’s cleanup when it has determined that the test has completed.
In order to wait for the full cleanup, change the last line from
executorService.shutdown();
to
executorService.shutdown();
executorService.awaitTermination(1, TimeUnit.DAYS);
Note that it is simpler to declare unexpected exceptions in the method’s throws clause rather than cluttering your test code with catch clauses calling Assertions.fail. JUnit will report such exceptions as failure anyway.
Then, you can remove the entire sleep code.
It might be worth putting the ExecutorService management into #Before/#After or even #BeforeClass/#AfterClass methods, to keep the testing methods free of that, to focus on the actual tests.¹
¹ These were the JUnit 4 names. IIRC, the JUnit 5 names are like #BeforeEach/#AfterEach resp. #BeforeAll/#AfterAll
I have the following implementation:
final ExecutorService service = Executors.newFixedThreadPool(4);
final List<Callable> taskList = new LinkedList<>();
for (Path path : paths) {
final SomeCallable task = new SomeCallable(path);
taskList.add(task);
service.submit(task);
}
synchronized (this) {
try {
this.wait();
shutdown(service);
} catch (InterruptedException e) {
shutdown(service);
}
}
And the following shutdown(service) method:
protected void shutdown(final ExecutorService service) {
List<Runnable> abortedCallables = service.shutdownNow();
try {
if (!service.awaitTermination(2, TimeUnit.MINUTES)) {
System.out.println("timeout");
}
}
catch (InterruptedException e) {
// (Re-)cancel all Callables if current thread has been interrupted
service.shutdownNow();
// Preserve interrupt status
Thread.currentThread().interrupt();
}
}
My problem is the following: paths is a list of a lot of Path instances. All of these paths will undergo some processing within a SomeCallable instance. Usually, this thread waits until all paths have been successfully handled within their SomeCallable. Once all paths have been processed shutdown(service) will be called.
However, it may happen that this thread is interrupted and has to be shutdown immediately. Therefore, I call shutdown(service) within the catch-block. What is working is, that currently running SomeCallable-instances will be finished and none of the tasks within the ExecutorService's thread pool will be started. Nonetheless, awaitTermination always runs into the 2 minute timeout. Why is this?
I assumed that awaitTermination waits for currently active running tasks to terminate (hence, wait for a maximum of 4 tasks -> thread pool size). Two minutes - in my case - is more than enough time for the 4 tasks to finish. As by calling shutdownNow() no new tasks from the queue should be started I wonder why I still get the timeout.
Any ideas?
This question already has answers here:
ExecutorService that interrupts tasks after a timeout
(11 answers)
Closed 7 years ago.
This is just an example to explain my problem...
I am using ExecutorService with 20 active threads and 75K max queued items...
In my case, a normal task should not take more than 10 seconds, if it takes more time that means there's some problem with the task.
If all the threads are hung due to problematic tasks my RejectionHandler would restart the entire service.
I have two questions here:
I do not like the idea of restarting the service, instead if there's
way to detect hanging thread and we could just restart that hung
thread that would be great. I have gone through couple of articles to handle hung threads with ThreadManager but have not found anything
with ExecutorService.
I am very much fascinated about the Executors.newCachedThredPool()
because on peak days we are heavily loaded with incoming tasks, and
on other days they are very few. Any suggestions would be greatly
appreciated.
public class HangingThreadTest {
// ExecutorService executorService = Executors.newCachedThreadPool()
private static ExecutorService executorService = new ThreadPoolExecutor(10,
20, 5L, TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(75000));
public static void main(String... arg0) {
for (int i = 0; i < 50000; i++) {
executorService.submit(new Task());
}
}
}
/**
* Task to be completed
*/
class Task implements Runnable {
private static int count = 0;
#Override
public void run() {
count++;
if (count%5 == 0) {
try {
System.out.println("Hanging Thread task that needs to be reprocessed: "
+ Thread.currentThread().getName()+" count: "+count);
Thread.sleep(11000);
} catch (InterruptedException e) {
// Do something
}
}
else{
System.out.println("Normal Thread: "
+ Thread.currentThread().getName()+" count: "+count);
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
//Do something
}
}
}
}
There is no build-in mechanism in Executors framework that would help terminate a thread if it has been running for more than a threshold value.
But we can achieve this with some extra code as below:
Get the Future object returned by the executorService.submit(...);.
Future future = executorService.submit(new Task());
Call the get method on this future object to and make it wait only for threshold interval for task completion. Below, an example that is waits for only 2 secs.
try {
f.get(2, TimeUnit.SECONDS);
} catch (TimeoutException e) {
f.cancel(true);
} catch (Exception e) {}
The above code waits for 2 seconds for task completion it throws a TimeoutException if it doesn't get completed during that time. Subsequently we can call cancel method on the future object. This results in setting the interrupt flag in the thread that is executing the task.
Now the final change is, in the Task class code we need to check at necessary points (application dependent), whether the interrupt flag has been set to true using isInterrupted() method of Thread class. If interrupted==true, we can do the necessary clean up and return from the run method immediately. The critical piece here is to identify the necessary points in your Task class where you want to check for this interrupted flag.
This makes the thread available for processing next task.
You may have a look at this article, it was very helpful for me before when I was facing the same problem : Java Hanging Thread Detection
My code:
String[] torrentFiles = new File("/root/torrents/").list();
if(torrentFiles.length == 0 || torrentFiles == null)
{
System.exit(0);
}
ex = Executors.newFixedThreadPool(3);
for(String torrentFile : torrentFiles)
{
ex.submit(new DownloadTorrent("/root/torrents/" + torrentFile));
}
ex.shutdown();
try
{
ex.awaitTermination(30, TimeUnit.MINUTES);
}
catch(InterruptedException ex1)
{
Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex1);
}
But sometimes torrent downloading takes unknown time value and «awaitTermination» not works as I want. I need to stop all executed threads instantly after half an hour but as I know «awaitTermination» just only use interrupt() method which works only in loops or waiting. So timeout not works if this moment happens. So, how to?
Instant thread termination is never guaranteed, unless the thread checks periodically for isInterrupted() flag (or is waiting in interruptable method, i.e. which throws InterruptedException).
Consider implementing your worker threads in manner, when they check periodically for isInterrupted(). This may be something like that:
public void run() {
byte[] data;
do {
data = receiveDataChunk(timeout);
processData(data);
} while(!isInterrupted() && data != null);
}
ExecutorService.shutdownNow() will try to stop all the executing threads..
Here is a quote from javadoc
List<Runnable> shutdownNow()
Attempts to stop all actively
executing tasks, halts the processing
of waiting tasks, and returns a list
of the tasks that were awaiting
execution.
There are no guarantees
beyond best-effort attempts to stop
processing actively executing tasks.
For example, typical implementations
will cancel via Thread.interrupt(), so
if any tasks mask or fail to respond
to interrupts, they may never
terminate.
Since downloading a torrent probably involves blocking IO operations, simply calling cancel()/shutdownNow() won't be enough, because blocking IO operations are not guaranteed to terminate when their respective threads are interrupted.
You also need to close the underlying sockets in order to cancel blocking IO, see How to terminate a thread blocking on socket IO operation instantly?.
ExecutorService.submit(...) returns a Future<?> that has a cancel() method. You should keep track of these can call it when you want each task to stop.
Am Using this code i have created.
Its generating many pdf files from many html templates using wkhtmltopdf .
so i want to increase performance of creating handreds without keep client waiting, this is only one part of implementation.
About getListOfCallables its returning the correct optimal
threshold for # of threads to use in fixed pool creation.
So i cant handle having lots of un dead threads laying around it made my EC2
CPU 100% stuck.
I used :
shutdown()
shutdownNow() in else of await
shutdownNow() in exception part
List fileGenerationHtmlToPdfList = getListOfCallables(paths, name, options);
ExecutorService executorService = Executors.newFixedThreadPool(fileGenerationHtmlToPdfList.size());
List<Future<ArrayList<File>>> futures = null;
try {
futures = executorService.invokeAll(fileGenerationHtmlToPdfList);
try {
for(Future f: futures) {
files.addAll((ArrayList<File>)f.get());
}
} catch (InterruptedException ex) {
Logger.getLogger(FileUtil.class.getName()).log(Level.SEVERE, "Interrupted Exception " , ex);
} catch (ExecutionException ex) {
Logger.getLogger(FileUtil.class.getName()).log(Level.SEVERE, "Interrupted Exception " , ex);
}
} catch (InterruptedException ex) {
Logger.getLogger(FileUtil.class.getName()).log(Level.SEVERE, "Interrupted Exception " , ex);
}
executorService.shutdown();//try shutdown
try {
if (executorService.awaitTermination(5, TimeUnit.SECONDS)) {
Logger.getLogger(FileUtil.class.getName()).log(Level.SEVERE, "Done ShutDowned");
} else {
executorService.shutdownNow();
}
} catch (InterruptedException ex) {
executorService.shutdownNow();
Logger.getLogger(FileUtil.class.getName()).log(Level.SEVERE, "Interrupted Exception " , ex);
}
Now I have to stop threads from a pool. I am doing it such a way. It may be not a good idea. Comment, please, if so.
boolean isTerminated = mPoolThreads.isTerminated();
while (!isTerminated) {
mPoolThreads.shutdownNow();
isTerminated = mPoolThreads.isTerminated();
Log.i(Constants.LOG_TAG, "Stop threads: the threads are not terminated yet");
}
Log.w(Constants.LOG_TAG, "Stop threads: Terminated");