Develop Reference

RXJava Observable onNext emitted on different threads

If i try to emit onNext on different thread's, on subscribing it dosent catch the stream on next elements.
public static Observable<Student> getStudents()
{
return Observable.create(e -> {
for(int i : Arrays.asList(1,2,3))
{
Thread t = new Thread(() -> {
e.onNext(new Student("anirba", i));
});
t.start();
}
e.onComplete();
});
}
On sunbscribing to this observable i dont get any responseenter code here
Observable<Student> observer = getStudents();
observer.subscribe(i -> System.out.println(i));

You are creating three threads inside the create method and each is adding an object of Student to stream. The reason why you are not getting any output is all these three threads will be running independently and based on thread scheduler it will be executed. In your case, the onComplete() method might get called before all these three threads add data into the Observable stream. and on-call of onComplete, the stream will be closed and no more data will be accepted by the stream. To make it work just to the below changes, it should work as you are expecting.
public static Observable<Student> getStudents() {
return Observable.create(e -> {
for(int i : Arrays.asList(1,2,3)) {
Thread t = new Thread(() -> {
e.onNext(new Student("anirba", i));
});
t.start();
}
Thread.sleep(1000);
e.onComplete();
});
}

java.util.concurrent.Executor is the right API now for running tasks. To block the current thread you can use CountDownLatch and release it when all tasks are terminated.
ExecutorService executor = Executors.newCachedThreadPool(); // choose the right one
public Observable<Student> getStudents() {
return Observable.<Student>create(emitter -> {
List<Integer> source = Arrays.asList(1, 2, 3);
CountDownLatch latch = new CountDownLatch(source.size());
source
.forEach(i ->
executor.submit(() -> {
emitter.onNext(new Student("anirba", i));
latch.countDown();
}));
latch.await();
emitter.onComplete();
}).serialize();
}
And elsewhere don't forget to call shutdown() on the executor.
You may also add serialize() operator to avoid onnext() calls to overlap.
From the contract :
Observables must issue notifications to observers serially (not in
parallel). They may issue these notifications from different threads,
but there must be a formal happens-before relationship between the
notifications.
For the perpose of testing you can add Thread.sleep(x) to see your loggin. I've already answerd this before here
public static void main(String[] args) throws InterruptedException {
getStudents()
.subscribe(i -> System.out.println(i));
Thread.sleep(2000);
}

Use CompletableFuture to execute single or multiple Callables and avoid blocking

I execute a few callables through ThreadPoolExecutor. If thread list contains only 1 callable then I directly call call method of my CallableService. If list contains more than 1 callables then I execute all those threads in parallel via thread pool executor.
How can I achieve this with Java 8 CompletableFuture? And if future.get() is enhanced to avoid blocking, that will be a plus.
private static ThreadPoolExecutor myThreadPoolExecutor = new ThreadPoolExecutor(0, 100, 5L, TimeUnit.SECONDS, new SynchronousQueue<>());
public static void execute(List<Callable<Boolean>> threadList) throws Exception {
List<Future<Boolean>> futureList = null;
CallableService singleService = (CallableService) threadList.get(0);
if (1 == threadList.size()) {
singleService.call();
}
else {
try {
futureList = myThreadPoolExecutor.invokeAll(threadList);
}
catch (InterruptedException e) {
e.printStackTrace();
}
}
if (null != futureList) {
for (Future<Boolean> future : futureList) {
try {
future.get();
}
catch (Exception e) {
//do some calculations here and then throw exception
throw new Exception(e.getMessage(), e);
}
}
}
}

There is no need for CompletableFuture, as the way you use the ExecutorService is sufficient, though, there are some aspects of the code flow which could be improved. You fetch the first element, even when not needed, and you cast it to CallableService for no reason, as you can call the method via the Callable interface already. In the other branch you are catching InterruptedException and proceeding, so the caller would never know that not all jobs have been executed. And in a straight-forward code flow, you don't need to check the list for null:
public static void execute(List<Callable<Boolean>> threadList) throws Exception {
if(1 == threadList.size()) {
Callable<Boolean> singleService = threadList.get(0);
singleService.call();
}
else {
List<Future<Boolean>> futureList = myThreadPoolExecutor.invokeAll(threadList);
for(Future<Boolean> future : futureList) {
try {
future.get();
}
catch(Exception e) {
//do some calculations here and then throw exception
throw new Exception(e.getMessage(), e);
}
}
}
}
You could shorten it further to
public static void execute(List<Callable<Boolean>> threadList) throws Exception {
if(1 == threadList.size()) {
threadList.get(0).call();
}
else {
for(Future<Boolean> future : myThreadPoolExecutor.invokeAll(threadList)) {
try {
future.get();
}
catch(Exception e) {
//do some calculations here and then throw exception
throw new Exception(e.getMessage(), e);
}
}
}
}
But that's a matter of preferred coding style. But note that it caught my eye that in the single element case, you're not performing the same exception handling.
To use CompletableFuture, we need an adapter method, as the convenience method supplyAsync requires a Supplier instead of a Callable. Using a modified variant of this answer, we get
public static void execute(List<Callable<Boolean>> threadList) throws Exception {
if(1 == threadList.size()) {
threadList.get(0).call();
}
else {
CompletableFuture<?> all = CompletableFuture.allOf(
threadList.stream()
.map(c -> callAsync(c, myThreadPoolExecutor))
.toArray(CompletableFuture<?>[]::new));
try {
all.get();
}
catch(Exception e) {
//do some calculations here and then throw exception
throw new Exception(e.getMessage(), e);
}
}
}
public static <R> CompletableFuture<R> callAsync(Callable<R> callable, Executor e) {
CompletableFuture<R> cf = new CompletableFuture<>();
CompletableFuture.runAsync(() -> {
try { cf.complete(callable.call()); }
catch(Throwable ex) { cf.completeExceptionally(ex); }
}, e);
return cf;
}
So we have no invokeAll which takes care of submitting all jobs. We have to do this manually, either with a loop or a stream operation. On the other hand, we get a single future via allOf representing the completion status, exceptionally if at least one job failed.
Unlike invokeAll, which waits for the completion, allOf only returns the future so it is the all.get() call which waits for the completion. We could do other things before it or even use this property to always perform the first job in the caller thread:
public static void execute(List<Callable<Boolean>> threadList) throws Exception {
CompletableFuture<?> tail = CompletableFuture.allOf(
threadList.stream().skip(1)
.map(c -> callAsync(c, myThreadPoolExecutor))
.toArray(CompletableFuture<?>[]::new)),
head = callAsync(threadList.get(0), Runnable::run);
try {
head.get();
tail.get();
}
catch(Exception e) {
//do some calculations here and then throw exception
throw new Exception(e.getMessage(), e);
}
}
This will always call the first callable in the current thread, as Runnable::run used as Executor will perform the action immediately in the calling thread. But it's treated uniformly in all other aspects, especially the exception handling. When there is only one job, allOf invoke with an empty array will do nothing and return an already completed future, which will have the desired effect.

Future.isDone() tells us if the executor has finished processing the task. If the task is completed, it will return true otherwise, it returns false.
for (Future<Boolean> future : futureList) {
while(!future.isDone())
{
doSOmethingElse();
Thread.sleep(300);//Optional
}
try {
future.get();
}
catch (Exception e)
{
//do some calculations here and then throw exception
throw new Exception(e.getMessage(), e);
}
}
But we don't have to worry about that since we get to the point where get() is called after making sure that the task is finished.

I execute a few callables through ThreadPoolExecutor. If thread list contains only 1 callable then I directly call call method of my CallableService. If list contains more than 1 callables then I execute all those threads in parallel via thread pool executor.
I guess you have already implemented this part. (You might run into memory usage issues if your jobs are heavy and you have 100 threads running as configured. But that is a different problem.)
And if future.get() is enhanced to avoid blocking, that will be a plus.
For this, you may take this approach:
Create another ExecutorService whose job will be just to run the Future.get() calls.
Submit your Future.get() to that service as shown below.
Shut it down and await termination.
if (null != futureList) {
ExecutorService waitSvc = Executors.newCachedThreadPool();
for (Future<Boolean> future : futureList) {
try {
waitSvc.submit( () -> future.get() );
}
catch (Exception e) {
//do some calculations here and then throw exception
throw new Exception(e.getMessage(), e);
}
}
waitSvc.shutdown(); //This may take some time. You may want to call awaitTermination() after this.
}
However, I feel that you should redesign the overall approach of using so many threads, unless this is only a for-learning application.

Detecting a timeout exception on a Java Future without calling get() on it

I am building a library that needs to some bluetooth operations on Android. I want to return a Future instance, so whoever is using my library can call .get() on the future returned and can handle ExecutionException, TimeoutException and InterruptedException themselves. However, I want to detect a timeout myself because I need to some cleanup logic like disconnecting from the device and so on. How can I achieve this?

You could implement a wrapper class around Future which delegates to a different one (the one returned by wherever you're getting your Future at the moment). Something like:
final class DelegatingFuture<T> implements Future<T> {
private final Future<T> delegate;
DelegatingFuture(final Future<T> delegate) {
this.delegate = Objects.requireNonNull(delegate);
}
// All other methods simply delegate to 'delegate'
#Override
public T get()
throws InterruptedException, ExecutionException {
try {
return this.delegate.get();
} catch (final Exception ex) {
// Handle cleanup...
throw ex;
}
}
// Something similar for get(long timeout, TimeUnit unit)
}
And then simply return new DelegatingFuture<>(currentFuture); wherever your handing these out.

The timeout is relevant to the caller of the get method with timeout and only to that caller. A timeout is nowhere meant to imply a cancellation. E.g., the following code is a legitimate usage of the Future API:
ExecutorService es = Executors.newSingleThreadExecutor();
Future<String> f = es.submit(() -> {
Thread.sleep(3000);
return "hello";
});
for(;;) try {
String s = f.get(500, TimeUnit.MILLISECONDS);
System.out.println("got "+s);
break;
}
catch(TimeoutException ex) {
// perhaps, do some other work
System.out.println("will wait something more");
}
catch (ExecutionException ex) {
System.out.println("failed with "+ex);
break;
}
es.shutdown();
Tying the cleanup to the methods actually intended to query the result, is not a useful approach. The timeout provided by the caller(s) of that method do not relate to the actual operation. There’s not even a guaranty that the result will be queried before the operations ends or that it gets queried at all.
The cleanup should happen when either, the operation finished or when the future gets cancelled explicitly. If the caller intends a cancellation after a timeout, the caller only needs to invoke cancel after catching a TimeoutException.
One approach, often pointed to, is to use a CompletionService, e.g.
static final ExecutorService MY__EXECUTOR = Executors.newCachedThreadPool();
static final CompletionService<String> COMPLETION_SERVICE
= new ExecutorCompletionService<>(MY__EXECUTOR);
static final Future<?> CLEANER = MY__EXECUTOR.submit(() -> {
for(;;) try {
Future<String> completed = COMPLETION_SERVICE.take();
System.out.println("cleanup "+completed);
} catch(InterruptedException ex) {
if(MY__EXECUTOR.isShutdown()) break;
}
});
public static Future<String> doSomeWork() {
return COMPLETION_SERVICE.submit(() -> {
Thread.sleep(3000);
return "hello";
});
}
You are in control over when to poll the completed futures, like in another background thread, as shown in the example, or right before commencing new jobs.
You can test it like
Future<String> f = doSomeWork();
try {
String s = f.get(500, TimeUnit.MILLISECONDS);
System.out.println("got "+s);
}
catch(TimeoutException ex) {
System.out.println("no result after 500ms");
}
catch (ExecutionException ex) {
System.out.println("failed with "+ex);
}
if(f.cancel(true)) System.out.println("canceled");
f = doSomeWork();
// never calling get() at all
But honestly, I never understood why such complicated things are actually necessary. If you want a cleanup at the right time, you can use
static final ExecutorService MY__EXECUTOR = Executors.newCachedThreadPool();
public static Future<String> doSomeWork() {
Callable<String> actualJob = () -> {
Thread.sleep(3000);
return "hello";
};
FutureTask<String> ft = new FutureTask<>(actualJob) {
#Override
protected void done() {
System.out.println("cleanup "+this);
}
};
MY__EXECUTOR.execute(ft);
return ft;
}
to achieve the same.
Or even simpler
static final ExecutorService MY__EXECUTOR = Executors.newCachedThreadPool();
public static Future<String> doSomeWork() {
Callable<String> actualJob = () -> {
Thread.sleep(3000);
return "hello";
};
return MY__EXECUTOR.submit(() -> {
try {
return actualJob.call();
}
finally {
// perform cleanup
System.out.println("cleanup");
}
});
}
In either case, the cleanup will be performed whether the job was completed successfully, failed, or got canceled. If cancel(true) was used and the actual job supports interruption, the cleanup also will be performed immediately after.

Vertx: executeBlocking() vs Future. What's the difference?

Vertx docs suggests to use executeBlocking() method when one needs to call blocking API. On the other hand Vertx also offers a notion of Future which basically do the same thing. But the executeBlocking() method isn't static. It is also not a simple wrapper around Future, and if you look at its implementation you'll see that it's pretty complex. What's the difference between these two?
Assume that I want to execute some long running task in an async way. Is there any difference between these two methods?
method 1:
doTheJob() {
Future<Void> future = Future.future();
executeLongRunningBlockingOperation();
future.complete();
return future;
}
doTheJob().setHandler(asyncResult -> {
// ... handle result
});
method 2:
vertx.executeBlocking(future -> {
executeLongRunningBlockingOperation();
future.complete();
}, res -> {
// ... handle result
});

Your first example is not a correct usage of Future. The call to executeLongRunningBlockingOperation() will block the main thread until that method has completed — i.e. nothing else can happen until the blocking operation finishes. In your second example the blocking call is spun off into a background thread and other things continue to happen while it executes.
To illustrate this with a more complete example, this code:
public void executeLongRunningBlockingOperation() {
Thread.sleep(5000);
}
public Future<Void> doTheJob() {
System.out.println("Doing the job...");
Future<Void> future = Future.future();
executeLongRunningBlockingOperation();
// this line will not be called until executeLongRunningBlockingOperation returns!
future.complete();
// nor will this method! This means that the method won't return until the long operation is done!
return future;
}
public static void main(String[] args) {
doTheJob().setHandler(asyncResult -> {
System.out.println("Finished the job");
});
System.out.println("Doing other stuff in the mean time...");
}
Will produce the following output:
Doing the job...
Finished the job
Doing other stuff in the mean time...
Whereas this code (using the executeBlocking):
...
public Future<Void> doTheJob() {
System.out.println("Doing the job...");
Future<Void> future = Future.future();
Vertx vertx = Vertx.vertx();
vertx.executeBlocking(call -> {
executeLongRunningBlockingOperation();
call.complete;
}, result -> {
// this will only be called once the blocking operation is done
future.complete();
});
// this method returns immediately since we are not blocking the main thread
return future;
}
...
Will produce:
Doing the job...
Doing other stuff in the mean time...
Finished the job
If you'd like to develop a better understanding of Vert.x I'd recommend the following hands-on tutorials:
https://vertx.io/docs/guide-for-java-devs/
http://escoffier.me/vertx-hol/

Run code after all Tasks of ThreadPoolExecuter are completed

I have a method in which I create a some files using ThreadPoolExecuter, and later zip the files created.
private void createAndZip(){
// Some Code
ThreadPoolExecutor executer = (ThreadPoolExecutor) Executors.newFixedThreadPool(5);
for(String str : someStringList){
// This piece of code creates files and drops to certain location.
executer.execute(new MyRunnable());
}
executer.shutdown();
// Code to Zip the files created above.
}
Now my piece of code to create zip files runs even before all files are created, so not all files are zipped.
Please help. I tried Sleep, but can't gaurantee how much time the files creation will take.

You need to invoke awaitTermination on the executor object, in order to wait for the executor to finish shutting down.

I used a CountDownLatch to solve the problem. Here is sample code.
private void createAndZip() throws Exception{
CountDownLatch latch = new CountDownLatch(someStringList.size());
// Some Code
ThreadPoolExecutor executer = (ThreadPoolExecutor) Executors.newFixedThreadPool(5);
for(String str : someStringList){
// This piece of code creates files and drops to certain location.
executer.execute(new MyRunnable(latch));
}
executer.shutdown();
// Code to Zip the files created above.
try {
latch.await();
} catch (InterruptedException exception) {
throw new GIException(exception);
}
//Code here.
}
public class MyRunnable implements Runnable{
CountDownLatch latch = null;
MyRunnable(CountDownLatch latch){
this.latch = latch;
}
#Override
public void run() {
try {
// Some Logic
latch.countDown();
} catch (Exception e) {
e.printStackTrace();
}
}
}

I think you can use Future objects here. Instead of calling execute() on the executor use submit() method. This should give you a Future object for each task you are submitting to the executor. Once you submit all tasks just loop over the list of futures you got and call get() on each. This is a blocking call and it waits until the corresponding task finishes.
Here the advantage is that you can retrieve any exception thrown from your task and then decide whether to zip the files or not.
Please refer this code -
private void createAndZip() throws Exception {
// Some Code
ThreadPoolExecutor executer = (ThreadPoolExecutor) Executors.newFixedThreadPool(5);
// collect all futures
List<Future> futures = new ArrayList<>();
for(String str : someStringList){
// This piece of code creates files and drops to certain location.
futures.add(executer.submit(new MyRunnable()));
}
// wait for all tasks to finish
try {
for (Future future : futures) {
future.get();
}
} catch (Exception e) {
e.printStackTrace();
if (e instanceof ExecutionException) {
throw e;
}
} finally {
executer.shutdown();
}
// Code to Zip the files created above.
}

In your code block you are narrowing the return of Executors.newFixedThreadPool(5). One option you have is to use the ExecutorService it returns. This class already has facilities that avoid having to re-implement synchronization code such as latches. For example:
Using Futures
private void createAndZip(ExecutorService executor) throws ExecutionException, InterruptedException {
// Some Code
List<String> list = new ArrayList<>();
// For a number of reasons ExecutorService should be constructed outside
// ExecutorService executer = Executors.newFixedThreadPool(5);
List<Future<?>> futures = new ArrayList<>();
for(String str : list){
// This piece of code creates files and drops to certain location.
futures.add(executer.submit(new MyRunnable()));
}
// async work
for (Future<?> future : futures) {
future.get(); // blocks
}
// Code to Zip the files created above.
}
There are some advantages here:
Error management: when executing the the background if you use another technique you have to arrange for errors to be delivered from the background thread to your master thread. Here the future takes care of this. If your worker throws then the exception will back it back to your controlling thread.
Keeping few threadpools in your code. The reason to pool threads in the first place is to make the startup costs smaller. If you have any significant sized program you wouldn't want to create and destroy threadpools whenever you wanted to perform an operation in parallel.
With Java8 Lambda this the loops can be written in a more compact way.
Fork/Join
Perhaps better suited to your task, particularly if you are going to process a tree of files is the Fork/Join framework. Here you could roll the processing and the zipping into a collection of tasks that are submitted to the fork-join pool. That's neat because you can get a Future for the whole zip file allowing you to produce the entire zip off your main thread. Something similar to your design using fork/join might be:
static class PrepareFile extends RecursiveTask<Void> {
private String filePath;
PrepareFile(String filePath) {
this.filePath = filePath;
}
#Override
protected Void compute() {
try {
System.out.println(filePath);
Thread.sleep(1009L);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
return null; // void
}
}
static class ZipTask extends RecursiveTask<String>
{
private List<String> files;
ZipTask(List<String> files) {
this.files = files;
}
#Override
protected String compute() {
List<PrepareFile> prepareTasks = new ArrayList<>();
for(String file : files) {
PrepareFile fileTask = new PrepareFile(file);
prepareTasks.add(fileTask);
fileTask.fork();
}
for(PrepareFile task : prepareTasks) {
task.join(); // can collect results here
}
System.out.println("Zipping");
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Done task");
return "filename.zip";
}
}
public static void main(String[] args) {
ForkJoinPool pool = new ForkJoinPool();
List<String> toProcess = Arrays.asList("a","b");
String filename = pool.invoke(new ZipTask(toProcess));
System.out.println("Zipped " + filename);
}
This is an illustration you'd want to change a few things, like the the return types of the tasks and how the tasks are invoked perhaps.
On awaitTermination
It is possible to use the awaitTermination method after calling shutdown to wait for all processes to terminate. However this may not be so desirable in longer running services or programs where thread-pools may be shared between operations.

private void createAndZip() throws Exception{
// Some Code
ThreadPoolExecutor executer = (ThreadPoolExecutor) Executors.newFixedThreadPool(5);
for(String str : someStringList){
// This piece of code creates files and drops to certain location.
executer.execute(new MyRunnable());
}
executer.shutdown();
while (true) {
boolean result_ = threadPoolExecutor.awaitTermination(TimeUnit.DAYS, 1);
if(result_)
break;
}
// Code to Zip the files created above.
//Code here.
}