My code:
String[] torrentFiles = new File("/root/torrents/").list();
if(torrentFiles.length == 0 || torrentFiles == null)
{
System.exit(0);
}
ex = Executors.newFixedThreadPool(3);
for(String torrentFile : torrentFiles)
{
ex.submit(new DownloadTorrent("/root/torrents/" + torrentFile));
}
ex.shutdown();
try
{
ex.awaitTermination(30, TimeUnit.MINUTES);
}
catch(InterruptedException ex1)
{
Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex1);
}
But sometimes torrent downloading takes unknown time value and «awaitTermination» not works as I want. I need to stop all executed threads instantly after half an hour but as I know «awaitTermination» just only use interrupt() method which works only in loops or waiting. So timeout not works if this moment happens. So, how to?
Instant thread termination is never guaranteed, unless the thread checks periodically for isInterrupted() flag (or is waiting in interruptable method, i.e. which throws InterruptedException).
Consider implementing your worker threads in manner, when they check periodically for isInterrupted(). This may be something like that:
public void run() {
byte[] data;
do {
data = receiveDataChunk(timeout);
processData(data);
} while(!isInterrupted() && data != null);
}
ExecutorService.shutdownNow() will try to stop all the executing threads..
Here is a quote from javadoc
List<Runnable> shutdownNow()
Attempts to stop all actively
executing tasks, halts the processing
of waiting tasks, and returns a list
of the tasks that were awaiting
execution.
There are no guarantees
beyond best-effort attempts to stop
processing actively executing tasks.
For example, typical implementations
will cancel via Thread.interrupt(), so
if any tasks mask or fail to respond
to interrupts, they may never
terminate.
Since downloading a torrent probably involves blocking IO operations, simply calling cancel()/shutdownNow() won't be enough, because blocking IO operations are not guaranteed to terminate when their respective threads are interrupted.
You also need to close the underlying sockets in order to cancel blocking IO, see How to terminate a thread blocking on socket IO operation instantly?.
ExecutorService.submit(...) returns a Future<?> that has a cancel() method. You should keep track of these can call it when you want each task to stop.
Am Using this code i have created.
Its generating many pdf files from many html templates using wkhtmltopdf .
so i want to increase performance of creating handreds without keep client waiting, this is only one part of implementation.
About getListOfCallables its returning the correct optimal
threshold for # of threads to use in fixed pool creation.
So i cant handle having lots of un dead threads laying around it made my EC2
CPU 100% stuck.
I used :
shutdown()
shutdownNow() in else of await
shutdownNow() in exception part
List fileGenerationHtmlToPdfList = getListOfCallables(paths, name, options);
ExecutorService executorService = Executors.newFixedThreadPool(fileGenerationHtmlToPdfList.size());
List<Future<ArrayList<File>>> futures = null;
try {
futures = executorService.invokeAll(fileGenerationHtmlToPdfList);
try {
for(Future f: futures) {
files.addAll((ArrayList<File>)f.get());
}
} catch (InterruptedException ex) {
Logger.getLogger(FileUtil.class.getName()).log(Level.SEVERE, "Interrupted Exception " , ex);
} catch (ExecutionException ex) {
Logger.getLogger(FileUtil.class.getName()).log(Level.SEVERE, "Interrupted Exception " , ex);
}
} catch (InterruptedException ex) {
Logger.getLogger(FileUtil.class.getName()).log(Level.SEVERE, "Interrupted Exception " , ex);
}
executorService.shutdown();//try shutdown
try {
if (executorService.awaitTermination(5, TimeUnit.SECONDS)) {
Logger.getLogger(FileUtil.class.getName()).log(Level.SEVERE, "Done ShutDowned");
} else {
executorService.shutdownNow();
}
} catch (InterruptedException ex) {
executorService.shutdownNow();
Logger.getLogger(FileUtil.class.getName()).log(Level.SEVERE, "Interrupted Exception " , ex);
}
Now I have to stop threads from a pool. I am doing it such a way. It may be not a good idea. Comment, please, if so.
boolean isTerminated = mPoolThreads.isTerminated();
while (!isTerminated) {
mPoolThreads.shutdownNow();
isTerminated = mPoolThreads.isTerminated();
Log.i(Constants.LOG_TAG, "Stop threads: the threads are not terminated yet");
}
Log.w(Constants.LOG_TAG, "Stop threads: Terminated");
Related
After reading several SO posts on how to kill a Java thread, I fairly understand why stop is unsafe and how to handle the graceful stop.
But the solutions are targeting towards UI threads where repainting is the problem and not really a long running - blocking process executed by a thread.
Links:
How do you kill a Thread in Java?
https://docs.oracle.com/javase/1.5.0/docs/guide/misc/threadPrimitiveDeprecation.html
One precise point that I fail to understand from the solutions or examples is what is the long-running-part the samples are trying to simulate.
Eg: In this following code, what if I set the interval to INT.MAX.
public void run() {
Thread thisThread = Thread.currentThread();
while (blinker == thisThread) {
try {
thisThread.sleep(interval); // This might take forever to complete,
// and while may never be executed 2nd time.
synchronized(this) {
while (threadSuspended && blinker==thisThread)
wait();
}
} catch (InterruptedException e){
}
repaint();
}
}
public synchronized void stop() {
blinker = null;
notify();
}
The reason am asking for this use case is that, I have a bug in a legacy code base that runs another executable in a Thread.
Now the ask if the user wishes to stop the thread, we would need to kill this thread, and the executable which is part of this thread automatically gets killed.
The way you stop a thread is by asking it - nicely - to stop. It's up to the code the thread is running to listen for and act on that request.
Specifically, the way you do it is to interrupt the thread. Your code checks for the interruption - Thread.sleep and Object.wait will throw InterruptedException if the thread is interrupted before or during their execution; but you catch the interruption, and ignore it, so you won't act on it.
Instead of this:
while (condition) {
try {
Thread.sleep(...);
wait();
} catch (InterruptedException e) {
}
}
Put the interruption outside the loop:
try {
while (condition) {
Thread.sleep(...);
wait();
}
} catch (InterruptedException e) {
}
then the loop terminates if it is interrupted.
Our Java application is having an issue where it blocks indefinitely when it tries to write to a log file located on a NFS share and the NFS share is down.
I was wondering whether we could solve this problem by having a Future execute the write operation with a timeout. Here is a little test program I wrote:
public class write_with_future {
public static void main(String[] args) {
int iteration=0;
while (true) {
System.out.println("iteration " + ++iteration);
ExecutorService executorService = Executors.newSingleThreadExecutor();
Future future = executorService.submit(new Runnable() {
public void run() {
try {
Category fileLogCategory = Category.getInstance("name");
FileAppender fileAppender = new FileAppender(new SimpleLayout(), "/usr/local/app/log/write_with_future.log");
fileLogCategory.addAppender(fileAppender);
fileLogCategory.log(Priority.INFO, System.currentTimeMillis());
fileLogCategory.removeAppender(fileAppender);
fileAppender.close();
}
catch (IOException e) {
System.out.println("IOException: " + e);
}
}
});
try {
future.get(100L, TimeUnit.MILLISECONDS);
}
catch (InterruptedException ie) {
System.out.println("Current thread interrupted while waiting for task to complete: " + ie);
}
catch (ExecutionException ee) {
System.out.println("Exception from task: " + ee);
}
catch (TimeoutException te) {
System.out.println("Task timed out: " + te);
}
finally {
future.cancel(true);
}
executorService.shutdownNow();
}
}
}
When I ran this program with a maximum heap size of 1 MB, and the NFS share was up, this program was able to execute over 1 million iterations before I stopped it.
But when I ran the program with a maximum heap size of 1 MB, and the NFS share was down, the program executed 584 iterations, getting a TimeoutException each time, and then it failed with a java.lang.OutOfMemoryError error. So I am thinking that even though future.cancel(true) and executorService.shutdownNow() are being called, the executor threads are blocked on the write and not responding to the interrupts, and the program eventually runs out of memory.
Is there any way to clean up the executor threads that are blocked?
If appears that Thread.interrupt() does not interrupt threads blocked in an I/O operation on an NFS file. You might want check the NFS mount options, but I suspect that you won't be able to fix that problem.
However, you could certainly prevent it from causing OOME's. The reason you are getting those is that you are not using ExecutorServices as they are designed to be used. What you are doing is repeatedly creating and shutting down single thread services. What you should be doing is creating on instance with a bounded thread pool and using that for all of the tasks. If you do it that way, if one of the threads takes a long time ... or is blocked in I/O ... you won't get a build-up of threads, and run out of memory. Instead, the backlogged tasks will sit in the ExecutorService's work queue until one of the worker thread unblocks.
I want to use the Executor interface (using Callable) in order to start a Thread (let's call it callable Thread) which will do work that uses blocking methods.
That means the callable Thread can throw an InterruptedException when the main Thread calls the Future.cancel(true) (which calls a Thread.interrupt()).
I also want my callable Thread to properly terminate when interrupted USING other blocking methods in a cancellation part of code.
While implementing this, I experienced the following behavior: When I call Future.cancel(true) method, the callable Thread is correctly notified of the interruption BUT if the main Thread immediately waits for its termination using Future.get(), the callable Thread is kind of killed when calling any blocking method.
The following JUnit 5 snippet illustrates the problem.
We can easily reproduce it if the main Thread does not sleep between the cancel() and the get() calls.
If we sleep a while but not enough, we can see the callable Thread doing half of its cancellation work.
If we sleep enough, the callable Thread properly completes its cancellation work.
Note 1: I checked the interrupted status of the callable Thread: it is correctly set once and only once, as expected.
Note 2: When debugging step by step my callable Thread after interruption (when passing into the cancellation code), I "loose" it after several step when entering a blocking method (no InterruptedException seems to be thrown).
#Test
public void testCallable() {
ExecutorService executorService = Executors.newSingleThreadExecutor();
System.out.println("Main thread: Submitting callable...");
final Future<Void> future = executorService.submit(() -> {
boolean interrupted = Thread.interrupted();
while (!interrupted) {
System.out.println("Callable thread: working...");
try {
Thread.sleep(500);
} catch (InterruptedException e) {
System.out.println("Callable thread: Interrupted while sleeping, starting cancellation...");
Thread.currentThread().interrupt();
}
interrupted = Thread.interrupted();
}
final int steps = 5;
for (int i=0; i<steps; ++i) {
System.out.println(String.format("Callable thread: Cancelling (step %d/%d)...", i+1, steps));
try {
Thread.sleep(200);
} catch (InterruptedException e) {
Assertions.fail("Callable thread: Should not be interrupted!");
}
}
return null;
});
final int mainThreadSleepBeforeCancelMs = 2000;
System.out.println(String.format("Main thread: Callable submitted, sleeping %d ms...", mainThreadSleepBeforeCancelMs));
try {
Thread.sleep(mainThreadSleepBeforeCancelMs);
} catch (InterruptedException e) {
Assertions.fail("Main thread: interrupted while sleeping.");
}
System.out.println("Main thread: Cancelling callable...");
future.cancel(true);
System.out.println("Main thread: Cancelable just cancelled.");
// Waiting "manually" helps to test error cases:
// - Setting to 0 (no wait) will prevent the callable thread to correctly terminate;
// - Setting to 500 will prevent the callable thread to correctly terminate (but some cancel process is done);
// - Setting to 1500 will let the callable thread to correctly terminate.
final int mainThreadSleepBeforeGetMs = 0;
try {
Thread.sleep(mainThreadSleepBeforeGetMs);
} catch (InterruptedException e) {
Assertions.fail("Main thread: interrupted while sleeping.");
}
System.out.println("Main thread: calling future.get()...");
try {
future.get();
} catch (InterruptedException e) {
System.out.println("Main thread: Future.get() interrupted: Error.");
} catch (ExecutionException e) {
System.out.println("Main thread: Future.get() threw an ExecutionException: Error.");
} catch (CancellationException e) {
System.out.println("Main thread: Future.get() threw an CancellationException: OK.");
}
executorService.shutdown();
}
When you call get() on a canceled Future, you will get a CancellationException, hence will not wait for the Callable’s code to perform its cleanup. Then, you are just returning and the observed behavior of threads being killed seems to be part of JUnit’s cleanup when it has determined that the test has completed.
In order to wait for the full cleanup, change the last line from
executorService.shutdown();
to
executorService.shutdown();
executorService.awaitTermination(1, TimeUnit.DAYS);
Note that it is simpler to declare unexpected exceptions in the method’s throws clause rather than cluttering your test code with catch clauses calling Assertions.fail. JUnit will report such exceptions as failure anyway.
Then, you can remove the entire sleep code.
It might be worth putting the ExecutorService management into #Before/#After or even #BeforeClass/#AfterClass methods, to keep the testing methods free of that, to focus on the actual tests.¹
¹ These were the JUnit 4 names. IIRC, the JUnit 5 names are like #BeforeEach/#AfterEach resp. #BeforeAll/#AfterAll
I think mistakenly guys compared take() vs poll(), but I found that it is reasonable to compare take() vs poll(time, unit) as both provided by BlockingQueue and both are blocking tell queue not Empty "and in case or poll or time-out", OK lets start comparing, usually I'm using take() for BlockingQueue but I was facing issues about:
handling interrupt inside loop.
waiting till be interrupted from outside.
how to stop looping on queue "using Kill-Bill or interrupt thread"
specially when work with Java 8 streams, then I got idea about I need to stop retrieving data from queue and close it in better way, so I thought to make waiting for sometime after that I can stop retrieve data then I found poll(time, unit) and it will fit for this idea check code below:
public static void main(String[] args) throws InterruptedException {
BlockingQueue<Integer> q = new LinkedBlockingQueue<Integer>();
ExecutorService executor = Executors.newCachedThreadPool();
executor.submit(() -> {
IntStream.range(0, 1000).boxed().forEach(i -> {
try {
q.put(i);
} catch (InterruptedException e) {
currentThread().interrupt();
throw new RuntimeException(e);
}
});
});
....
// Take
Future fTake = executor.submit(() -> {
try {
while (!Thread.currentThread().isInterrupted()) {
System.out.println(q.take());
}
} catch (InterruptedException e) {
currentThread().interrupt();
throw new RuntimeException(e);
}
});
//to stop it I have to do below code "Expecting that execution will take 1 sec"
executor.shutdown();
sleep(1000);
fTake.cancel(true);
....
// poll there is no need to expect time till processing will be done
Future fPoll = executor.submit(() -> {
try {
Integer i;
while ((i = q.poll(100, TimeUnit.MILLISECONDS)) != null)
System.out.println(i);
} catch (InterruptedException e) {
currentThread().interrupt();
throw new RuntimeException(e);
}
});
executor.shutdown();
}
I think the poll code is more clean and there is no need to depend on interrupt and also no need to estimate execution time or make code to determined when to interrupt thread, what do you think?
Note 1: I'm sure that 2nd solution also have drawbacks like not getting data till time-out but I think you are going to know what is suitable time-out for your case.
Note 2: if use case requires waiting for ever and producer is low frequency provide data, I think take solution is better.
The main process:
int cpus = Runtime.getRuntime().availableProcessors();
ExecutorService executor = Executors.newFixedThreadPool(cpus);
List<Callable<Object>> todo = new ArrayList<Callable<Object>>(lines.size());
for (int r = 0; r < lines.size(); r++) {
String filename = r + 1 + "";
todo.add(Executors.callable(new AppConsole(filename, lines.get(r))));
}
List<Future<Object>> answers = executor.invokeAll(todo);
The AppConsole class implements Runnable and the overriden run method is as follows:
public void run() {
try{
} catch (SecurityException exception) {
exception.printStackTrace();
} catch (FileNotFoundException exception) {
exception.printStackTrace();
} catch (IOException exception) {
exception.printStackTrace();
} catch (SAXException exception) {
exception.printStackTrace();
} catch (Exception exception) {
exception.printStackTrace();
} finally {
if (output != null) {
output.close();
}
}
}
}
The main process is suspended and cannot finish because of a connection timeout exception thrown from one of the threads. Now I can see the cpu usage drops to 0% and the memory consumption keeps at the same level when it got suspended.
Could someone please help me solve this problem? At least, help me make the main process finish?
Throwing an exception frees up the tasks and the main thread. The ExecutorService treats an Exception throw much like another return value and doesn't have a problem handling it.
The main thread will only block waiting for one of your tasks to complete. I would look at the tasks/threads which are still active to see what they are doing e.g. perhaps they haven't timed out yet.
you could use the alternative version of invokeAll that takes an explicit timeout value :
http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/ExecutorService.html
invokeAll(Collection<? extends Callable<T>> tasks,
long timeout,
TimeUnit unit)
this way you can force all of your tasks to timeout after, say, 30 seconds. Then you can inspect the Futures returned to see which have completed.
You just need to add:
executor.shutdown();
This waits when all tasks will be completed and shutdown all threads, application will exit after that.
Please see javadoc for newFixedThreadPool:
Creates a thread pool that reuses a fixed number of threads operating
off a shared unbounded queue. At any point, at most nThreads threads
will be active processing tasks. If additional tasks are submitted
when all threads are active, they will wait in the queue until a
thread is available. If any thread terminates due to a failure during
execution prior to shutdown, a new one will take its place if needed
to execute subsequent tasks. The threads in the pool will exist until
it is explicitly shutdown.