Develop Reference

Run a script for a specified period of time in Java

I have a java code which I want to run. If the job does not complete within, say, 2 hours, then it should be killed automatically (basically, some kind of timed batch).
How to achieve this in Java?

If you are on Java 9 or higher, you can do the timeout batching as below:-
CompletableFuture<Integer> future = CompletableFuture.supplyAsync(this::longRunningTask)
.orTimeout(2, TimeUnit.SECONDS);
future.get(); // j.u.c.ExecutionException after waiting for 2 second
If it completes within the timeout limit, it will return the value (here an Integer object in response to future.get() method)
And, this batching is asynchronous (If you don't call get method explicitly.)
NOTE: This does not prevent the thread from completing the task, it just completes a future in main thread with a Timeout Exception so that main thread can continue. The background task/thread is still continues to finish. (look #Andreas comment)
Some Samples:-
final CompletableFuture<Void> future =
CompletableFuture.supplyAsync(this::longRunningTask)
.orTimeout(2, TimeUnit.SECONDS);
future.get(); // j.u.c.ExecutionException after waiting for 2 second
And the longRunningTask() :-
private Void longRunningTask(){
log.info("Thread name" + Thread.currentThread().getName());
try {
log.info("Going to sleep for 10 sec...");
Thread.sleep(10*1000);
log.info("Sleep Completed. Task Completed.");
} catch (InterruptedException e) {
log.info("Exception Occurred");
}
finally{
log.info("Final Cleanup in progress.");
}
log.info("Finishing the long task.");
return null;
}
If you run above code, it will give Execution Exception in main thread (where future.get() is called) but the longRunningTask will still print Sleep Completed. Task Completed. after completing 10 sec sleep.
If you carefully notice, the longRunnigThread is never interrupted (does not enter in catch block) so continues normally, but main thread gets an exception on get().
Workaround/Solution:
Use ExecutorService and submit the longRunnigTask future with this Exceutor, if timeout occurs, shutdown the executor or else, shutdown after successful get() in case of no timeout exception.
Sample:
ExecutorService myWorkers = Executors.newFixedThreadPool(1);
final CompletableFuture<Void> longTask =
CompletableFuture.supplyAsync(this::longRunningTask, myWorkers)
.orTimeout(2, TimeUnit.SECONDS);
try {
longTask.get();
} catch (InterruptedException | ExecutionException e) {
log.info("EE... Kill the executor thread/s.");
myWorkers.shutdownNow(); // this will interrupt the thread, catch the IntrExcep in thread and return the execution there
}
and the slightly modified longRunnigTask
private Void longRunningTask(){
log.info("Thread name" + Thread.currentThread().getName());
try {
log.info("Going to sleep for 10 sec...");
Thread.sleep(10*1000);
log.info("Sleep Completed. Task Completed.");
} catch (InterruptedException e) {
log.info("Exception Occurred");
return null; // this will finish the thread exce releasing all locking resources. Can be GCed then.
}
finally{
log.info("Final Cleanup in progress.");
}
log.info("Finishing the long task.");
return null;
}
With this approach, it won't complete the task inf timeout is occurred (you won't see Sleep completed. Task completed. in logs..), and would see, exception occurred in the longRunningTask thread (because of interrupt caused by myWorker.shutdown).

Thread - Can't exit while loop on ExecutorService.isShutDown

So my threads are working as expected, and I just wanted to add some extra sauce to my homework.
I made a while loop that checks uses the isShutdown which returns false unless shutdown(); has been called.
So i call shutdown at the end of my method, but it won't ever exit the while loop.
public void runParrallel() throws InterruptedException {
System.out.println("Submitting Task ...");
ExecutorService executor = Executors.newFixedThreadPool(5);
List<Future<TagCounter>> counters = new ArrayList();
counters.add(executor.submit(new TagCounterCallable("https//www.fck.dk")));
counters.add(executor.submit(new TagCounterCallable("https://www.google.com")));
counters.add(executor.submit(new TagCounterCallable("https://politiken.dk")));
counters.add(executor.submit(new TagCounterCallable("https://cphbusiness.dk")));
System.out.println("Task is submitted");
while (!executor.isShutdown()) {
System.out.println("Task is not completed yet....");
Thread.sleep(1000);
}
for (Future<TagCounter> future : counters) {
try {
TagCounter tc = future.get();
System.out.println("Title: " + tc.getTitle());
System.out.println("Div's: " + tc.getDivCount());
System.out.println("Body's: " + tc.getBodyCount());
System.out.println("----------------------------------");
} catch (ExecutionException ex) {
System.out.println("Exception: " + ex);
}
}
executor.shutdown();
}

The while-loop is before you ever call shutdown(). The condition cannot possibly evaluate to false, thus you are stuck with an infinite loop. I'd suggest moving the while loop to the point after you call shutdown().
See also this question on how to shut down an ExecutorService.

Correct me if I'm wrong, but it looks like you want to wait until all tasks that were submitted to your ExecutorService have finished. If you know that they're going to finish in a timely manner, then you can use ExecutorService#shutdown in conjunction with ExecutorService#awaitTermination to block the executing thread until all tasks are complete.
This can be done with the following:
public void runParrallel() throws InterruptedException {
// Same code to submit tasks.
System.out.println("Task is submitted");
executor.shutdown();
executor.awaitTermination(1, TimeUnit.DAYS);
// At this point, the ExecutorService has been shut down successfully
// and all tasks have finished.
for (Future<TagCounter> future : counters) {
try {
TagCounter tc = future.get();
System.out.println("Title: " + tc.getTitle());
System.out.println("Div's: " + tc.getDivCount());
System.out.println("Body's: " + tc.getBodyCount());
System.out.println("----------------------------------");
} catch (ExecutionException ex) {
System.out.println("Exception: " + ex);
}
}
}
With this solution, the while loop can be removed.

Your while-loop is running infinitely because there is nothing that activates the executor.shutdown() inside the while-loop. The code wont progress to the end where you call executor.shutdown() because the while-loop's condition returns back to the start of the while-loop.
Put an if-statement inside the while-loop. The if-statement checks if the task is submitted, and if it is, the executor.shutdown() will be called.
Following is just an example:
while (!executor.isShutdown()) {
System.out.println("Task is not completed yet....");
Thread.sleep(1000);
if(TaskIsCompleted){
executor.shutdown();
}
}

Receiving items emitted from an Observable

Here's my code:
public static void main(String[] args) {
Observable.just("747", "737", "777")
.flatMap(
a -> {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
return Observable.just(a).subscribeOn(Schedulers.newThread());
})
.subscribe(p -> System.out.println("Received " + p + " on thread " + Thread.currentThread().getName()));
}
As I have understood it, each of the items of the observable should will run in a separate thread (which does happen) and the results would be sent to the same thread that did the work (this happens as well). But what I cannot understand is that why is the main thread not exiting and waiting for the background threads to finish? The program continues for as long as each of the background threads is running.

If you look at a thread dump, you'll see that the main thread is actually stuck on the sleep statement. That's why it's not exiting.
This is because it is the thread that's executing the flatMap operator, so it's getting stuck. This also is why the code takes a long time to run. You can easily verify it by inserting a print statement just before sleep:
try {
System.out.println(Thread.currentThread().getName() + " is sleeping");
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
The output is something like this:
main is sleeping
main is sleeping
Received 747 on thread RxNewThreadScheduler-1
main is sleeping
Received 737 on thread RxNewThreadScheduler-2
Received 777 on thread RxNewThreadScheduler-3
I think you've meant to write something like this:
System.out.println(Thread.currentThread().getName() + " is creating the observable");
Observable.just("747", "737", "777")
.flatMap(a ->
Observable.fromCallable(() -> {
try {
System.out.println(Thread.currentThread().getName() + " is sleeping");
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
return a;
}).subscribeOn(Schedulers.newThread())
).subscribe(p -> System.out.println("Received " + p + " on thread " + Thread.currentThread().getName()));
System.out.println(Thread.currentThread().getName() + " is going to exit");
Output:
main is creating the observable
main is going to exit
RxNewThreadScheduler-3 is sleeping
RxNewThreadScheduler-2 is sleeping
RxNewThreadScheduler-1 is sleeping
Received 777 on thread RxNewThreadScheduler-3
Received 747 on thread RxNewThreadScheduler-1
Received 737 on thread RxNewThreadScheduler-1
In this version main is exiting right after the Observable is created.

Enforce executorService.awaitTermination while using CompletionService

I am trying to submit multiple tasks and obtain the results as and when it is available. However, after the end of the loop, I have to enforce that all the tasks complete within specified amount of time. If not, throw an error. Initially, all I had was executorService's invokeAll, shutdown and awaitTermination calls that were used to ensure that all tasks complete (inspite of errors or not). I migrated the code to use CompletionService to display the results. Where can I enforce the awaitTermination clause in the CompletionService calls?
CompletionService<String> completionService = new ExecutorCompletionService<String>(executor);
logger.info("Submitting all tasks");
for (Callable<String> task : tasks)
completionService.submit(task);
executor.shutdown();
logger.info("Tasks submitted. Now checking the status.");
while (!executor.isTerminated())
{
final Future<String> future = completionService.take();
String itemValue;
try
{
itemValue = future.get();
if (!itemValue.equals("Bulk"))
logger.info("Backup completed for " + itemValue);
}
catch (InterruptedException | ExecutionException e)
{
String message = e.getCause().getMessage();
String objName = "Bulk";
if (message.contains("(") && message.contains(")"))
objName = message.substring(message.indexOf("(") + 1, message.indexOf(")"));
logger.error("Failed retrieving the task status for " + objName, e);
}
}
executor.awaitTermination(24, TimeUnit.HOURS);
In other words, how can I utilize timeout for CompletionService?
EDIT:
The initial code I had was displayed below. The problem is that I am iterating through the future list and then printing them as completed. However, my requirement is to display the ones that were completed at a FCFS basis.
List<Future<String>> results = executor.invokeAll(tasks);
executor.shutdown();
executor.awaitTermination(24, TimeUnit.HOURS);
while (results.size() > 0)
{
for (Iterator<Future<String>> iterator = results.iterator(); iterator.hasNext();)
{
Future<String> item = iterator.next();
if (item.isDone())
{
String itemValue;
try
{
itemValue = item.get();
if (!itemValue.equals("Bulk"))
logger.info("Backup completed for " + itemValue);
}
catch (InterruptedException | ExecutionException e)
{
String message = e.getCause().getMessage();
String objName = "Bulk";
if (message.contains("(") && message.contains(")"))
objName = message.substring(message.indexOf("(") + 1, message.indexOf(")"));
logger.error("Failed retrieving the task status for " + objName, e);
}
finally
{
iterator.remove();
}
}
}
}

I'd suggest you wait for the executor to terminate on another thread
That way you can achieve serving results FCFS and also enforce the timeout.
It can be easily achieved with something that will look like the following
CompletionService<String> completionService = new ExecutorCompletionService<String>(executor);
// place all the work in a function (an Anonymous Runnable in this case)
// completionService.submit(() ->{work});
// as soon as the work is submitted it is handled by another Thread
completionService.submit(() ->{
logger.info("Submitting all tasks");
for (Callable<String> task : tasks)
completionService.submit(task);
logger.info("Tasks submitted. Now checking the status.");
int counter = tasks.size();
for(int i = counter; counter >=1; counter--) // Replaced the while loop
{
final Future<String> future = completionService.take();
String itemValue;
try
{
itemValue = future.get();
if (!itemValue.equals("Bulk"))
logger.info("Backup completed for " + itemValue);
}
catch (InterruptedException | ExecutionException e)
{
String message = e.getCause().getMessage();
String objName = "Bulk";
if (message.contains("(") && message.contains(")"))
objName = message.substring(message.indexOf("(") + 1, message.indexOf(")"));
logger.error("Failed retrieving the task status for " + objName, e);
}
}
});
// After submitting the work to another Thread
// Wait in your Main Thread, and enforce termination if needed
shutdownAndAwaitTermination(executor);
You handle the executors termination && waiting using this (taken from ExecutorsService)
void shutdownAndAwaitTermination(ExecutorService pool) {
pool.shutdown(); // Disable new tasks from being submitted
try {
// Wait a while for existing tasks to terminate
if (!pool.awaitTermination(24, TimeUnit.HOURS)) {
pool.shutdownNow(); // Cancel currently executing tasks
// Wait a while for tasks to respond to being cancelled
if (!pool.awaitTermination(60, TimeUnit.SECONDS))
System.err.println("Pool did not terminate");
}
} catch (InterruptedException ie) {
// (Re-)Cancel if current thread also interrupted
pool.shutdownNow();
// Preserve interrupt status
Thread.currentThread().interrupt();
}
}

Ok then, you need to monitor completion. So, why are yon not using as per documentation? https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ExecutorCompletionService.html So, it submits n tasks to a new instance of ExecutorCompletionService and waits n to complete. No termination again, you could just reuse the same executor (usually thread pool, creating a new thread is more expensive rather than reusing from a pool). So, if I adapt code from the documentation to your scenario it would be something like:
CompletionService<Result> ecs
= new ExecutorCompletionService<String>(executor);
for (Callable<Result> task : tasks)
ecs.submit(task);
logger.info("Tasks submitted. Now checking the status.");
int n = tasks.size();
for (int i = 0; i < n; ++i) {
try {
String r = ecs.take().get();
logger.info("Backup completed for " + r);
}
catch(InterruptedException | ExecutionException e) {
...
}
}
Also, it is bad idea to parse exception message, better if you create your custom exception class and use instanceof.
If you need to have a timeout for the completion - use poll with time parameters instead of take.

Cleaning up after two threads finish execution

I have an application that is running jobs that require two threads for every job. The two threads normally do some work and finish shortly after each other. Then after the second thread finishes I need to do some cleanup but since the threads are doing some network IO, it is possible for one thread to get blocked for a long time. In that case, I want the cleanup to take place a few seconds after the first thread finishes.
I implemented this behaviour with the following piece of code in a callback class:
private boolean first = true;
public synchronized void done() throws InterruptedException {
if (first) {
first = false;
wait(3000);
// cleanup here, as soon as possible
}
else {
notify();
}
}
Both threads invoke the done() method when they finish. The first one will then block in the wait() for at most 3 seconds but will be notified immediately when the seconds thread invokes the done() method.
I have tested this implementation and it seems to work well but I'm am curious if there's a better way of doing this. Even though this implementation doesn't look too complicated, I'm afraid that my program will deadlock or have some unsuspected synchronization issue.

I hope I understood your need. You want to wait for thread-a to complete and then wait either 3 seconds or for the end of thread-b.
It is better to use the newer Concurrent tools instead of the old wait/notify as there are so many edge cases to them.
// Two threads running so count down from 2.
CountDownLatch wait = new CountDownLatch(2);
class TestRun implements Runnable {
private final long waitTime;
public TestRun(long waitTime) {
this.waitTime = waitTime;
}
#Override
public void run() {
try {
// Wait a few seconds.
Thread.sleep(waitTime);
// Finished! Count me down.
wait.countDown();
System.out.println(new Date() + ": " + Thread.currentThread().getName() + " - Finished");
} catch (InterruptedException ex) {
System.out.println(Thread.currentThread().getName() + " - Interrupted");
}
}
}
public void test() throws InterruptedException {
// ThreadA
Thread threadA = new Thread(new TestRun(10000), "Thread A");
// ThreadB
Thread threadB = new Thread(new TestRun(30000), "Thread B");
// Fire them up.
threadA.start();
threadB.start();
// Wait for all to finish but threadA must finish.
threadA.join();
// Wait up to 3 seconds for B.
wait.await(3, TimeUnit.SECONDS);
System.out.println(new Date() + ": Done");
threadB.join();
}
happily prints:
Tue Sep 15 16:59:37 BST 2015: Thread A - Finished
Tue Sep 15 16:59:40 BST 2015: Done
Tue Sep 15 16:59:57 BST 2015: Thread B - Finished
Added
With the new clarity - that the end of any thread starts the timer - we can use a third thread for the cleanup. Each thread must call a method when it finishes to trigger the cleanup mechanism.
// Two threads running so count down from 2.
CountDownLatch wait = new CountDownLatch(2);
class TestRun implements Runnable {
private final long waitTime;
public TestRun(long waitTime) {
this.waitTime = waitTime;
}
#Override
public void run() {
try {
// Wait a few seconds.
Thread.sleep(waitTime);
// Finished! Count me down.
wait.countDown();
System.out.println(new Date() + ": " + Thread.currentThread().getName() + " - Finished");
// Record that I've finished.
finished();
} catch (InterruptedException ex) {
System.out.println(Thread.currentThread().getName() + " - Interrupted");
}
}
}
Runnable cleanup = new Runnable() {
#Override
public void run() {
try {
// Wait up to 3 seconds for both threads to clear.
wait.await(3, TimeUnit.SECONDS);
// Do your cleanup stuff here.
// ...
System.out.println(new Date() + ": " + Thread.currentThread().getName() + " - Finished");
} catch (InterruptedException ex) {
System.out.println(Thread.currentThread().getName() + " - Interrupted");
}
}
};
final AtomicBoolean cleanupStarted = new AtomicBoolean(false);
private void finished() {
// Make sure I only start the cleanup once.
if (cleanupStarted.compareAndSet(false, true)) {
new Thread(cleanup, "Cleanup").start();
}
}
public void test() throws InterruptedException {
// ThreadA
Thread threadA = new Thread(new TestRun(10000), "Thread A");
// ThreadB
Thread threadB = new Thread(new TestRun(30000), "Thread B");
// Fire them up.
threadA.start();
threadB.start();
System.out.println(new Date() + ": Done");
}

As done method is synchronized, so only one thread can execute at a time, with this second will wait to send notify until first finishes its whole job, which might cause performance bottleneck.
I would rather design it with short synchronized block which would primarily update the boolean first.