Develop Reference

Ways to use ExecutorService to handle a task completion using multiple threads

I have an executor service which creates, say, 10 threads. Each thread is supposed to handle processing of some piece of data. Finally, we have to wait for all the 10 threads to finish processing.
First approach
This is how I am currently implementing this:
public Processor implements Callable<Void> {
private Data myData;
public Processor(Data myData) {
this.myData = myData;
}
public Void call() throws BusinessLogicException {
// Business logic to handle myData
}
}
Handler class to create & handle threads:
public class Handler() {
ExecutorService executorService;
public Handler() {
executorService = Executors.newFixedThreadPool(10);
}
public void handle(List<Data> dataList) {
List<Processor> workers = new ArrayList<>();
List<String> dataId = new ArrayList<>();
for(Data data : dataList) {
workers.add(new Processor(data));
dataId.add(data.getId());
}
try {
List<Future<Void>> futures = executorService.invokeAll(workers);
int i = 0;
for(Future<Void> future : futures) {
try {
future.get();
} catch(CancellationException | InterruptedException | ExecutionException e) {
System.err.println("For data with id " + dataId.get(i) + " encountered " + e);
} finally {
i++;
}
}
} catch(InterruptedException | NullPointerException | RejectedExecutionException e) {
System.err.println(e);
} finally {
executorService.shutdown();
}
}
}
Second approach
Another way I can think of is to write callable as a lambda expression. Something of the sorts:
List<Future<Void>> futures = new ArrayList<>();
for(Data myData: dataList) {
workers.add(
executorService.submit( () -> {
try {
//Business logic to handle myData
} catch (Exception e) {
//Log the error using myData and e
//Need to handle all exceptions here, not just BusinessLogicException
}
})
)
}
for(Future<Void> future : futures) {
//The same try-catch logic as above
}
I need feedback on which is the better and cleaner code design to handle this problem.
Issues I see with first approach:
Multiple instances of Processor created. Processor is stateful because it need input Data to process - I can't seem to think of a good way to circumvent this issue.
I need to maintain a List dataId which I use to identify the ID of the data which a thread failed to process.
Issues I see with second approach:
Seems messy compared to first one.
Need to handle all the exceptions within the lambda expression.

Detecting a timeout exception on a Java Future without calling get() on it

I am building a library that needs to some bluetooth operations on Android. I want to return a Future instance, so whoever is using my library can call .get() on the future returned and can handle ExecutionException, TimeoutException and InterruptedException themselves. However, I want to detect a timeout myself because I need to some cleanup logic like disconnecting from the device and so on. How can I achieve this?

You could implement a wrapper class around Future which delegates to a different one (the one returned by wherever you're getting your Future at the moment). Something like:
final class DelegatingFuture<T> implements Future<T> {
private final Future<T> delegate;
DelegatingFuture(final Future<T> delegate) {
this.delegate = Objects.requireNonNull(delegate);
}
// All other methods simply delegate to 'delegate'
#Override
public T get()
throws InterruptedException, ExecutionException {
try {
return this.delegate.get();
} catch (final Exception ex) {
// Handle cleanup...
throw ex;
}
}
// Something similar for get(long timeout, TimeUnit unit)
}
And then simply return new DelegatingFuture<>(currentFuture); wherever your handing these out.

The timeout is relevant to the caller of the get method with timeout and only to that caller. A timeout is nowhere meant to imply a cancellation. E.g., the following code is a legitimate usage of the Future API:
ExecutorService es = Executors.newSingleThreadExecutor();
Future<String> f = es.submit(() -> {
Thread.sleep(3000);
return "hello";
});
for(;;) try {
String s = f.get(500, TimeUnit.MILLISECONDS);
System.out.println("got "+s);
break;
}
catch(TimeoutException ex) {
// perhaps, do some other work
System.out.println("will wait something more");
}
catch (ExecutionException ex) {
System.out.println("failed with "+ex);
break;
}
es.shutdown();
Tying the cleanup to the methods actually intended to query the result, is not a useful approach. The timeout provided by the caller(s) of that method do not relate to the actual operation. There’s not even a guaranty that the result will be queried before the operations ends or that it gets queried at all.
The cleanup should happen when either, the operation finished or when the future gets cancelled explicitly. If the caller intends a cancellation after a timeout, the caller only needs to invoke cancel after catching a TimeoutException.
One approach, often pointed to, is to use a CompletionService, e.g.
static final ExecutorService MY__EXECUTOR = Executors.newCachedThreadPool();
static final CompletionService<String> COMPLETION_SERVICE
= new ExecutorCompletionService<>(MY__EXECUTOR);
static final Future<?> CLEANER = MY__EXECUTOR.submit(() -> {
for(;;) try {
Future<String> completed = COMPLETION_SERVICE.take();
System.out.println("cleanup "+completed);
} catch(InterruptedException ex) {
if(MY__EXECUTOR.isShutdown()) break;
}
});
public static Future<String> doSomeWork() {
return COMPLETION_SERVICE.submit(() -> {
Thread.sleep(3000);
return "hello";
});
}
You are in control over when to poll the completed futures, like in another background thread, as shown in the example, or right before commencing new jobs.
You can test it like
Future<String> f = doSomeWork();
try {
String s = f.get(500, TimeUnit.MILLISECONDS);
System.out.println("got "+s);
}
catch(TimeoutException ex) {
System.out.println("no result after 500ms");
}
catch (ExecutionException ex) {
System.out.println("failed with "+ex);
}
if(f.cancel(true)) System.out.println("canceled");
f = doSomeWork();
// never calling get() at all
But honestly, I never understood why such complicated things are actually necessary. If you want a cleanup at the right time, you can use
static final ExecutorService MY__EXECUTOR = Executors.newCachedThreadPool();
public static Future<String> doSomeWork() {
Callable<String> actualJob = () -> {
Thread.sleep(3000);
return "hello";
};
FutureTask<String> ft = new FutureTask<>(actualJob) {
#Override
protected void done() {
System.out.println("cleanup "+this);
}
};
MY__EXECUTOR.execute(ft);
return ft;
}
to achieve the same.
Or even simpler
static final ExecutorService MY__EXECUTOR = Executors.newCachedThreadPool();
public static Future<String> doSomeWork() {
Callable<String> actualJob = () -> {
Thread.sleep(3000);
return "hello";
};
return MY__EXECUTOR.submit(() -> {
try {
return actualJob.call();
}
finally {
// perform cleanup
System.out.println("cleanup");
}
});
}
In either case, the cleanup will be performed whether the job was completed successfully, failed, or got canceled. If cancel(true) was used and the actual job supports interruption, the cleanup also will be performed immediately after.

Editable queue of tasks running in background thread

I know this question was answered many times, but I'm struggling to understand how it works.
So in my application the user must be able to select items which will be added to a queue (displayed in a ListView using an ObservableList<Task>) and each item needs to be processed sequentially by an ExecutorService.
Also that queue should be editable (change the order and remove items from the list).
private void handleItemClicked(MouseEvent event) {
if (event.getClickCount() == 2) {
File item = listView.getSelectionModel().getSelectedItem();
Task<Void> task = createTask(item);
facade.getTaskQueueList().add(task); // this list is bound to a ListView, where it can be edited
Future result = executor.submit(task);
// where executor is an ExecutorService of which type?
try {
result.get();
} catch (Exception e) {
// ...
}
}
}
Tried it with executor = Executors.newFixedThreadPool(1) but I don't have control over the queue.
I read about ThreadPoolExecutor and queues, but I'm struggling to understand it as I'm quite new to Concurrency.
I need to run that method handleItemClicked in a background thread, so that the UI does not freeze, how can I do that the best way?
Summed up: How can I implement a queue of tasks, which is editable and sequentially processed by a background thread?
Please help me figure it out
EDIT
Using the SerialTaskQueue class from vanOekel helped me, now I want to bind the List of tasks to my ListView.
ListProperty<Runnable> listProperty = new SimpleListProperty<>();
listProperty.set(taskQueue.getTaskList()); // getTaskList() returns the LinkedList from SerialTaskQueue
queueListView.itemsProperty().bind(listProperty);
Obviously this doesn't work as it's expecting an ObservableList. There is an elegant way to do it?

The simplest solution I can think of is to maintain the task-list outside of the executor and use a callback to feed the executor the next task if it is available. Unfortunately, it involves synchronization on the task-list and an AtomicBoolean to indicate a task executing.
The callback is simply a Runnable that wraps the original task to run and then "calls back" to see if there is another task to execute, and if so, executes it using the (background) executor.
The synchronization is needed to keep the task-list in order and at a known state. The task-list can be modified by two threads at the same time: via the callback running in the executor's (background) thread and via handleItemClicked method executed via the UI foreground thread. This in turn means that it is never exactly known when the task-list is empty for example. To keep the task-list in order and at a known fixed state, synchronization of the task-list is needed.
This still leaves an ambiguous moment to decide when a task is ready for execution. This is where the AtomicBoolean comes in: a value set is always immediatly availabe and read by any other thread and the compareAndSet method will always ensure only one thread gets an "OK".
Combining the synchronization and the use of the AtomicBoolean allows the creation of one method with a "critical section" that can be called by both foreground- and background-threads at the same time to trigger the execution of a new task if possible. The code below is designed and setup in such a way that one such method (runNextTask) can exist. It is good practice to make the "critical section" in concurrent code as simple and explicit as possible (which, in turn, generally leads to an efficient "critical section").
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicBoolean;
public class SerialTaskQueue {
public static void main(String[] args) {
ExecutorService executor = Executors.newSingleThreadExecutor();
// all operations on this list must be synchronized on the list itself.
SerialTaskQueue tq = new SerialTaskQueue(executor);
try {
// test running the tasks one by one
tq.add(new SleepSome(10L));
Thread.sleep(5L);
tq.add(new SleepSome(20L));
tq.add(new SleepSome(30L));
Thread.sleep(100L);
System.out.println("Queue size: " + tq.size()); // should be empty
tq.add(new SleepSome(10L));
Thread.sleep(100L);
} catch (Exception e) {
e.printStackTrace();
} finally {
executor.shutdownNow();
}
}
// all lookups and modifications to the list must be synchronized on the list.
private final List<Runnable> tasks = new LinkedList<Runnable>();
// atomic boolean used to ensure only 1 task is executed at any given time
private final AtomicBoolean executeNextTask = new AtomicBoolean(true);
private final Executor executor;
public SerialTaskQueue(Executor executor) {
this.executor = executor;
}
public void add(Runnable task) {
synchronized(tasks) { tasks.add(task); }
runNextTask();
}
private void runNextTask() {
// critical section that ensures one task is executed.
synchronized(tasks) {
if (!tasks.isEmpty()
&& executeNextTask.compareAndSet(true, false)) {
executor.execute(wrapTask(tasks.remove(0)));
}
}
}
private CallbackTask wrapTask(Runnable task) {
return new CallbackTask(task, new Runnable() {
#Override public void run() {
if (!executeNextTask.compareAndSet(false, true)) {
System.out.println("ERROR: programming error, the callback should always run in execute state.");
}
runNextTask();
}
});
}
public int size() {
synchronized(tasks) { return tasks.size(); }
}
public Runnable get(int index) {
synchronized(tasks) { return tasks.get(index); }
}
public Runnable remove(int index) {
synchronized(tasks) { return tasks.remove(index); }
}
// general callback-task, see https://stackoverflow.com/a/826283/3080094
static class CallbackTask implements Runnable {
private final Runnable task, callback;
public CallbackTask(Runnable task, Runnable callback) {
this.task = task;
this.callback = callback;
}
#Override public void run() {
try {
task.run();
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
callback.run();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
// task that just sleeps for a while
static class SleepSome implements Runnable {
static long startTime = System.currentTimeMillis();
private final long sleepTimeMs;
public SleepSome(long sleepTimeMs) {
this.sleepTimeMs = sleepTimeMs;
}
#Override public void run() {
try {
System.out.println(tdelta() + "Sleeping for " + sleepTimeMs + " ms.");
Thread.sleep(sleepTimeMs);
System.out.println(tdelta() + "Slept for " + sleepTimeMs + " ms.");
} catch (Exception e) {
e.printStackTrace();
}
}
private String tdelta() { return String.format("% 4d ", (System.currentTimeMillis() - startTime)); }
}
}
Update: if groups of tasks need to be executed serial, have a look at the adapted implementation here.

How to release resource in canceled CompletableFuture

Uscase
Suppose we run execution with CompletableFuture.runAsync(..) and in runnable we have try-with-resources block (we are using some resource that should be closed whatever happens), and at some point when execution is not finished in try block we cancel the completable future... altough execution is stopped the resource that should be closed is not closed the close() of AutoClosable is not called...
Question
Is that a java issue or there is a way to do that properly? without hacky workarounds like using futures (that support interruption etc..), and if its expected behaviour how should one handle similar situation when non interruptable CompletableFuture is canceled...?
The Code
public class AutoClosableResourceTest {
public static class SomeService{
public void connect(){
System.out.println("connect");
}
public Integer disconnect(){
System.out.println("disconnect");
return null;
}
}
public static class AutoClosableResource<T> implements AutoCloseable {
private final T resource;
private final Runnable closeFunction;
private AutoClosableResource(T resource, Runnable closeFunction){
this.resource = resource;
this.closeFunction = closeFunction;
}
public T get(){
return resource;
}
#Override
public void close() throws Exception {
closeFunction.run();
}
}
#Test
public void testTryWithResource() throws InterruptedException {
SomeService service = new SomeService();
CompletableFuture<Void> async = CompletableFuture.runAsync(() -> {
try (AutoClosableResource<SomeService> resource = new AutoClosableResource<>(service, service::disconnect)) {
resource.get().connect();
while (true) {
Thread.sleep(1000);
System.out.println("working...");
}
} catch (Exception e) {
e.printStackTrace();
}
});
Thread.sleep(2500);
async.cancel(true);
Thread.sleep(2500);
}
}
this will produce
connect
working...
working...
working...
working...
as you can see it does not call cancel() and leaves resource opened...

It seems you have difficulties in understanding what the purpose of CompletableFuture is. Have a look at the first sentence of its class documentation:
A Future that may be explicitly completed (setting its value and status), …
So unlike FutureTask which is completed by the thread executing its run method, a CompletableFuture can be completed by any thread which will set its value/status at an arbitrary point of time. The CompletableFuture doesn’t know which thread will complete it and it doesn’t even know whether there is a thread currently working on its completion.
Therefore the CompletableFuture can not interrupt the right thread when being canceled. That’s a fundamental part of its design.
If you want a worker thread that you can interrupt you are better off using FutureTask/ThreadPoolExecutor. The task scheduled that way may still complete a CompletableFuture at its end.

The following code will be stuck in an infinite loop. Calling async.cancel will not communicate with the following loop its desire to stop.
while (true) {
Thread.sleep(1000);
System.out.println("working...");
}
The test case exits because the thread stuck in this loop is not a daemon thread.
Replace the while loop check with the following, which checks the isCancelled flag on each iteration. Calling CompletableFuture.cancel() will mark the future as cancelled, but it does not interrupt the thread that was started via runAsync.
while (isCancelled()) {
Thread.sleep(1000);
System.out.println("working...");
}

You could use the "complete" method of the CompletableFuture to stop the thread.
Below a simple code to show the behaviour:
package com.ardevco;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class CompletableFutureTest3 {
public static void main(String[] args) throws Exception {
ExecutorService pool = Executors.newFixedThreadPool(5);
CompletableFuture<Integer> longRunningcompletableFuture = CompletableFuture.supplyAsync(() -> {
for (int i = 0; i < 1; i--) {
System.out.println("i " + i);
sleep();
}
return 1; // we will newer reach this line so the thread will be stuck
});
CompletableFuture<Integer> completor = CompletableFuture.supplyAsync(() -> {
System.out.println("completing the longRunningcompletableFuture");
longRunningcompletableFuture.complete(1000);
System.out.println("completed the longRunningcompletableFuture");
return 10;
});
Thread.sleep(10000);
System.out.println("completor...");
int i = completor.get();
System.out.println("completor i:" + i);
System.out.println("completor...");
System.out.println("completableFutureToBeCompleted2...");
int i2 = longRunningcompletableFuture.get();
System.out.println("completableFutureToBeCompleted2: " + i2);
System.out.println("completableFutureToBeCompleted2...");
}
private static void sleep() {
try {Thread.sleep(1000);}catch (Exception e) {}
}
}
output:
i 0
completing the longRunningcompletableFuture
completed the longRunningcompletableFuture
i -1
i -2
i -3
i -4
i -5
i -6
i -7
i -8
i -9
i -10
completor...
completor i:10
completor...
completableFutureToBeCompleted2...
completableFutureToBeCompleted2: 1000
completableFutureToBeCompleted2...

Though there is an answer marked as correct, the reason is quite different - please see documentation for CompletableFuture.cancel(mayInterruptIfRunning) method and read the article CompletableFuture can't be interrupted to understand the problem better.
This issue is addressed in my Tascalate Concurrent library, the changes to your code should be:
From
CompletableFuture<Void> async = CompletableFuture.runAsync(() -> {
...
});
To
Promise<Void> async = CompletableTask.runAsync(() -> {
...
}, someExplicitExecutor);
...and you will get expected behavior (executor thread is interrupted, AutoClosable is closed, async is completed with CancellationException).
You can read more about the library in my blog

I also face this problem in Java 8 SE. For me, it's important not to use third-party libraries.
cancel(mayInterruptIfRunning) this value has no effect in this implementation because interrupts are not used to control processing.
The idea is to use Thread.interrupt() when calling cancel(), but only for Runnable.
/** Enable and disable the interrupt */
private static class Interruptor {
volatile boolean interrupted;
volatile Runnable interrupt;
/** Enable interrupt support */
synchronized boolean start() {
if (interrupted) {
return false;
}
Thread runThread = Thread.currentThread();
interrupt = () -> {
if (runThread != Thread.currentThread()) {
runThread.interrupt();
}
};
return true;
}
/** Interrupt Runnable */
synchronized void interrupt() {
if (interrupted) {
return;
}
interrupted = true;
if (interrupt != null) {
interrupt.run();
interrupt = null;
}
}
/** Disable interrupt support */
synchronized void finish() {
interrupt = null;
}
}
/** CompletableFuture with interrupt support */
public static CompletableFuture<Void> runAsyncInterrupted(Runnable run) {
final Interruptor interruptor = new Interruptor();
Runnable wrap = () -> {
if (!interruptor.start()) { // allow interruption
return; // was canceled before the thread started
}
try {
run.run(); // can be interrupted
} finally {
interruptor.finish(); // can no longer be interrupted
}
};
CompletableFuture<Void> cfRun = CompletableFuture.runAsync(wrap);
// here is caught "CompletableFuture.cancel()"
cfRun.whenComplete((r, t) -> {
if (t instanceof CancellationException) {
interruptor.interrupt();
}
});
return cfRun;
}
Example of use
Runnable mySlowIoRun = () -> {
try {
InputStream is = openSomeResource(); // open resource
try {
// there may be problem (#1) with reading,
// such as loss of network connection
int bt = is.read();
// ..
// .. some code
} finally {
is.close(); // problem (#2): releases any system resources associated with the stream
}
} catch (Throwable th) {
throw new RuntimeException(th);
}
};
CompletableFuture<Void> cf = runAsyncInterrupted(mySlowIoRun);
try {
cf.get(5, TimeUnit.SECONDS); // 5 sec timeout
} catch (Throwable th) {
cf.cancel(true); // cancel with interrupt mySlowIoRun
throw th;
}

So this is a generalization of how I typically handle the problem.. pass-in a cancellable state, and close resources IMMEDIATELY after the open state.
private static BufferedReader openFile(String fn) {
try {
return Files.newBufferedReader(Paths.get(fn));
} catch (IOException e) {
throw new RuntimeException(e);
}
}
static class Util {
static void closeQuietly(AutoCloseable c) {
if (c == null) return;
try {
c.close();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
static <T extends AutoCloseable, R> R runThenCloseQuietly(T c, Function<T,R> cb) {
try {
return cb.apply(c);
} finally {
closeQuietly(c);
}
}
static <T extends AutoCloseable, R> Optional<R> runThenCloseQuietlyCancellable(BooleanSupplier cancelled
, T c, Function<T,Optional<R>> cb) {
if (c == null) return Optional.empty(); // safe doesn't throw
try {
if (cancelled.getAsBoolean()) return Optional.empty(); // might throw, wrap for safety
return cb.apply(c); // might throw
} finally {
closeQuietly(c); // might throw, but at least we're closed
}
}
private static Optional<String> emptyString() {
return Optional.empty();
}
}
interface Cancellable {
boolean isCancelled();
void cancel();
}
static class CancellableAB implements Cancellable {
private final AtomicBoolean cancelled;
CancellableAB(AtomicBoolean cancelled) {
this.cancelled = cancelled;
}
#Override
public boolean isCancelled() {
return cancelled.get();
}
#Override
public void cancel() {
cancelled.set(true);
}
}
static class CancellableArray implements Cancellable {
private final boolean[] cancelled;
private final int idx;
CancellableArray(boolean[] cancelled) {
this(cancelled, 0);
}
CancellableArray(boolean[] cancelled, int idx) {
this.cancelled = cancelled;
this.idx = idx;
}
#Override
public boolean isCancelled() {
return cancelled[idx];
}
#Override
public void cancel() {
cancelled[idx]=true;
}
}
static class CancellableV implements Cancellable {
volatile boolean cancelled;
#Override
public boolean isCancelled() {
return cancelled;
}
#Override
public void cancel() {
this.cancelled = true;
}
}
/**
* The only reason this is a class is because we need SOME external object for the lambda to check for mutated
* cancelled state.
* This gives the added benefit that we can directly call cancel on the resource.
* We allow a cancellable to be passed in to CHAIN-IN cancellable state. e.g. if cancellation should affect MULTIPLE
* CompletableFuture states, we don't want other promises to tie references to this task.. So the cancellable
* object can be externalized.
*
* Normally you don't need this much genericism, you can directly implement a volatile 'cancel boolean'.
* But this allows you to create a C.F. task as a 3rd party library call - gives maximum flexibility to invoker.
*
*/
static class FooTask {
volatile Cancellable cancelled;
String fileName;
public FooTask(String fileName) {
this.fileName = fileName;
this.cancelled = new CancellableV();
}
public FooTask(String fileName, Cancellable cancelled) {
this.cancelled = cancelled;
}
public boolean isCancelled() {
return cancelled.isCancelled();
}
public void cancel() {
cancelled.cancel();
}
/**
* asynchronously opens file, scans for first valid line (closes file), then processes the line.
* Note if an exception happens, it's the same as not finding any lines. Don't need to special case.
* Use of utility functions is mostly for generic-mapping
* (avoiding annoying double-type-casting plus editor warnings)
*/
CompletableFuture<Optional<Long>> run1() {
return
CompletableFuture.supplyAsync(() -> openFile(fileName))
.thenApplyAsync(c -> { // this stage MUST close the prior stage
if(cancelled.isCancelled() || c == null) return Util.emptyString(); // shouldn't throw
try {
return c
.lines()
.filter(line -> !cancelled.isCancelled())
.filter(line -> !line.startsWith("#"))
.findFirst();
} catch (RuntimeException e) {
Util.closeQuietly(c);
throw new RuntimeException(e);
}
}
)
.thenApplyAsync(oLine -> // this stage doesn't need closing
oLine
.map(line -> line.split(":"))
.map(cols -> cols[2])
.map(Long::valueOf)
)
;
}
/**
* Same as run1 but avoids messy brackets + try-finally
*/
CompletableFuture<Optional<Long>> run2() {
return
CompletableFuture.supplyAsync(() -> openFile(fileName))
.thenApplyAsync(c -> // this stage MUST close the prior stage
Util.runThenCloseQuietly(
c
, r -> cancelled.isCancelled() ? Util.emptyString() // shouldn't throw
: r
.lines()
.filter(line -> !cancelled.isCancelled())
.filter(line -> !line.startsWith("#"))
.findFirst()
))
.thenApplyAsync(oLine -> // this stage doesn't need closing
oLine
.map(line -> line.split(":"))
.map(cols -> cols[2])
.map(Long::valueOf)
)
;
}
/**
* Same as run2 but avoids needing the teneary operator - says Cancellable in func-name so is more readable
*/
CompletableFuture<Optional<Long>> run3() {
return
CompletableFuture.supplyAsync(() -> openFile(fileName))
.thenApplyAsync(c -> // this stage MUST close the prior stage
Util.runThenCloseQuietlyCancellable(
cancelled::isCancelled // lambda here is slightly easier to read than explicit if-statement
, c
, r -> r
.lines()
.filter(line -> !cancelled.isCancelled())
.filter(line -> !line.startsWith("#"))
.findFirst()
))
.thenApplyAsync(oLine -> // this stage doesn't need closing
oLine
.map(line -> line.split(":"))
.map(cols -> cols[2])
.map(Long::valueOf)
)
;
}
}
#Test
public void testFooGood() {
var task = new FooTask("/etc/passwd");
var cf = task.run3();
var oVal = cf.join();
assertTrue(oVal.isPresent());
System.out.println(oVal.get()); // should not throw
}
#Test
public void testFooCancel() {
var task = new FooTask("/etc/passwd");
var cf = task.run3();
task.cancel();
var oVal = cf.join();
assertTrue(oVal.isEmpty());
}

Executing Dependent tasks in parallel in Java

I need to find a way to execute tasks (dependent and independent) in parallel in java.
Task A and Task C can run independently.
Task B is dependent on the output of Task A.
I checked java.util.concurrent Future and Fork/Join, but looks like we cannot add dependency to a Task.
Can anyone point me to correct Java API.

In Scala this is very easy to do, and I think you are better off using Scala. Here is an example I pulled from here http://danielwestheide.com/ (The Neophyte’s Guide to Scala Part 16: Where to Go From Here) this guy has a great blog (I am not that guy)
Lets take a barrista making coffee. The tasks to do are:
Grind the required coffee beans (no preceding tasks)
Heat some water (no preceding tasks)
Brew an espresso using the ground coffee and the heated water (depends on 1 & 2)
Froth some milk (no preceding tasks)
Combine the froth milk and the espresso (depends on 3,4)
or as a tree:
Grind _
Coffe \
\
Heat ___\_Brew____
Water \_____Combine
/
Foam ____________/
Milk
In java using the concurrency api this would be:
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.FutureTask;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
public class Barrista {
static class HeatWater implements Callable<String> {
#Override
public String call() throws Exception {
System.out.println("Heating Water");
Thread.sleep(1000);
return "hot water";
}
}
static class GrindBeans implements Callable<String> {
#Override
public String call() throws Exception {
System.out.println("Grinding Beans");
Thread.sleep(2000);
return "grinded beans";
}
}
static class Brew implements Callable<String> {
final Future<String> grindedBeans;
final Future<String> hotWater;
public Brew(Future<String> grindedBeans, Future<String> hotWater) {
this.grindedBeans = grindedBeans;
this.hotWater = hotWater;
}
#Override
public String call() throws Exception
{
System.out.println("brewing coffee with " + grindedBeans.get()
+ " and " + hotWater.get());
Thread.sleep(1000);
return "brewed coffee";
}
}
static class FrothMilk implements Callable<String> {
#Override
public String call() throws Exception {
Thread.sleep(1000);
return "some milk";
}
}
static class Combine implements Callable<String> {
public Combine(Future<String> frothedMilk, Future<String> brewedCoffee) {
super();
this.frothedMilk = frothedMilk;
this.brewedCoffee = brewedCoffee;
}
final Future<String> frothedMilk;
final Future<String> brewedCoffee;
#Override
public String call() throws Exception {
Thread.sleep(1000);
System.out.println("Combining " + frothedMilk.get() + " "
+ brewedCoffee.get());
return "Final Coffee";
}
}
public static void main(String[] args) {
ExecutorService executor = Executors.newFixedThreadPool(2);
FutureTask<String> heatWaterFuture = new FutureTask<String>(new HeatWater());
FutureTask<String> grindBeans = new FutureTask<String>(new GrindBeans());
FutureTask<String> brewCoffee = new FutureTask<String>(new Brew(grindBeans, heatWaterFuture));
FutureTask<String> frothMilk = new FutureTask<String>(new FrothMilk());
FutureTask<String> combineCoffee = new FutureTask<String>(new Combine(frothMilk, brewCoffee));
executor.execute(heatWaterFuture);
executor.execute(grindBeans);
executor.execute(brewCoffee);
executor.execute(frothMilk);
executor.execute(combineCoffee);
try {
/**
* Warning this code is blocking !!!!!!!
*/
System.out.println(combineCoffee.get(20, TimeUnit.SECONDS));
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
} catch (TimeoutException e) {
System.out.println("20 SECONDS FOR A COFFEE !!!! I am !##! leaving!!");
e.printStackTrace();
} finally{
executor.shutdown();
}
}
}
Make sure that you add time outs though to ensure that your code will not wait forever on something to complete, that is done by using the Future.get(long, TimeUnit) and then handle failure accordingly.
It is much nicer in scala however, here it is like it's on the blog:
The code to prepare some coffee would look something like this:
def prepareCappuccino(): Try[Cappuccino] = for {
ground <- Try(grind("arabica beans"))
water <- Try(heatWater(Water(25)))
espresso <- Try(brew(ground, water))
foam <- Try(frothMilk("milk"))
} yield combine(espresso, foam)
where all the methods return a future (typed future), for instance grind would be something like this:
def grind(beans: CoffeeBeans): Future[GroundCoffee] = Future {
// grinding function contents
}
For all the implementations check out the blog but that's all there is to it. You can integrate Scala and Java easily as well. I really recommend doing this sort of thing in Scala instead of Java. Scala requires much less code, much cleaner and event driven.

General programming model for tasks with dependencies is Dataflow. Simplified model where each task has only one, though repeating, dependency is Actor model. There are many actor libraries for Java, but very few for dataflow.
See also: which-actor-model-library-framework-for-java, java-pattern-for-nested-callbacks

Use a BlockingQueue. Put the output of task A into the queue, and task B blocks until something is available in the queue.
The docs contain example code to achieve this: http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/BlockingQueue.html

Java defines a class CompletableFuture.
https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html
This is what you are looking for.
It helps to build execution flows.

What you need is a CountDownLatch.
final CountDownLatch gate = new CountDownLatch(2);
// thread a
new Thread() {
public void run() {
// process
gate.countDown();
}
}.start();
// thread c
new Thread() {
public void run() {
// process
gate.countDown();
}
}.start();
new Thread() {
public void run() {
try {
gate.await();
// both thread a and thread c have completed
// process thread b
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}.start();
As an alternative, depending on your scenario, you might also be able to use a BlockingQueue to implement the Producer-Consumer pattern. See the example on the documentation page.

If task B is dependent on task A's output, I would first question whether or not task B really is a separate task. Separating the tasks would make sense if there is:
Some non-trivial amount of work that task B can do before needing task A's results
Task B is a long ongoing process that handles output from many different instances of task A
There is some other tasks (say D) that also use task A's results
Assuming it is a separate task, then you can allow task A & B to share a BlockingQueue such that task A can pass task B data.

Use this library https://github.com/familysyan/TaskOrchestration. It manages the task dependency for you.

There is a java library specifically for this purpose (Disclaimer : I am the owner of this library) called Dexecutor
Here is how you can achieve the desired result, you can read more about it here
#Test
public void testDependentTaskExecution() {
DefaultDependentTasksExecutor<String, String> executor = newTaskExecutor();
executor.addDependency("A", "B");
executor.addIndependent("C");
executor.execute(ExecutionBehavior.RETRY_ONCE_TERMINATING);
}
private DefaultDependentTasksExecutor<String, String> newTaskExecutor() {
return new DefaultDependentTasksExecutor<String, String>(newExecutor(), new SleepyTaskProvider());
}
private ExecutorService newExecutor() {
return Executors.newFixedThreadPool(ThreadPoolUtil.ioIntesivePoolSize());
}
private static class SleepyTaskProvider implements TaskProvider<String, String> {
public Task<String, String> provid(final String id) {
return new Task<String, String>() {
#Override
public String execute() {
try {
//Perform some task
Thread.sleep(500);
} catch (InterruptedException e) {
e.printStackTrace();
}
String result = id + "processed";
return result;
}
#Override
public boolean shouldExecute(ExecutionResults<String, String> parentResults) {
ExecutionResult<String, String> firstParentResult = parentResults.getFirst();
//Do some logic with parent result
if ("B".equals(id) && firstParentResult.isSkipped()) {
return false;
}
return true;
}
};
}
}