i'm trying to process some amount of data concurrently using CompletableFuture and Stream
So far i have:
public static void main(String[] args) throws InterruptedException, ExecutionException {
System.out.println("start");
List<String> collect = Stream.of("1", "2", "3", "4", "5",
"6", "7")
.map(x -> CompletableFuture.supplyAsync(getStringSupplier(x)))
.collect(Collectors.toList())
.stream()
.map(CompletableFuture::join)
.collect(Collectors.toList());
System.out.println("stop out!");
}
public static Supplier<String> getStringSupplier(String text) {
return () -> {
System.out.println("start " + text);
try {
TimeUnit.SECONDS.sleep(2);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("stop " + text);
return "asd" + text;
};
}
And output is fine :
start
start 1
start 4
start 3
start 2
start 5
start 6
start 7
stop 4
stop 1
stop 5
stop 2
stop 6
stop 3
stop 7
stop out!
However right now i want to add timeout to that job. Lets say it should be canceled after 1 SECOND. And return null or some other value to collect list. (I would prefer some value indicating cause).
How can i achieve that ?
Thanks for help in advance.
I have found the way of doing that:
private static final ScheduledExecutorService scheduler =
Executors.newScheduledThreadPool(
1,
new ThreadFactoryBuilder()
.setDaemon(true)
.setNameFormat("failAfter-%d")
.build());
public static void main(String[] args) throws InterruptedException, ExecutionException {
System.out.println("start");
final CompletableFuture<Object> oneSecondTimeout = failAfter(Duration.ofSeconds(1))
.exceptionally(xxx -> "timeout exception");
List<Object> collect = Stream.of("1", "2", "3", "4", "5", "6", "7")
.map(x -> CompletableFuture.anyOf(createTaskSupplier(x)
, oneSecondTimeout))
.collect(Collectors.toList())
.stream()
.map(CompletableFuture::join)
.collect(Collectors.toList());
System.out.println("stop out!");
System.out.println(collect);
}
public static CompletableFuture<String> createTaskSupplier(String x) {
return CompletableFuture.supplyAsync(getStringSupplier(x))
.exceptionally(xx -> "PROCESSING ERROR : " + xx.getMessage());
}
public static Supplier<String> getStringSupplier(String text) {
return () -> {
System.out.println("start " + text);
try {
TimeUnit.MILLISECONDS.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
if (text.equals("1")) {
throw new RuntimeException("LOGIC ERROR");
}
try {
if (text.equals("7"))
TimeUnit.SECONDS.sleep(2);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("stop " + text);
return "result " + text;
};
}
public static <T> CompletableFuture<T> failAfter(Duration duration) {
final CompletableFuture<T> promise = new CompletableFuture<>();
scheduler.schedule(() -> {
final TimeoutException ex = new TimeoutException("Timeout after " + duration);
return promise.completeExceptionally(ex);
}, duration.toMillis(), MILLISECONDS);
return promise;
}
It returns :
start
start 1
start 3
start 4
start 2
start 5
start 6
start 7
stop 6
stop 4
stop 3
stop 5
stop 2
stop out!
[PROCESSING ERROR : java.lang.RuntimeException: LOGIC ERROR, result 2, result 3, result 4, result 5, result 6, timeout exception]`
What do you think about that, can you spot any flaws of that solution ?
For others, who are not limited with Java 8, you can use completeOnTimeout method, which was introduced in Java 9.
List<String> collect = Stream.of("1", "2", "3", "4", "5", "6", "7")
.map(x -> CompletableFuture.supplyAsync(getStringSupplier(x))
.completeOnTimeout(null , 1, SECONDS))
.filter(Objects::nonNull)
.collect(toList())
.stream()
.map(CompletableFuture::join)
.collect(toList());
You can wrap the job in another CompletableFuture and it would give out a TimeoutException if the given time is exceeded. You can separate the TimeoutException catch block if you want to handle it specially.
List<String> collect = null;
try {
collect = CompletableFuture.supplyAsync(() ->
Stream.of("1", "2", "3", "4", "5",
"6", "7")
.map(x -> CompletableFuture.supplyAsync(getStringSupplier(x)))
.collect(Collectors.toList())
.stream()
.map(CompletableFuture::join)
.collect(Collectors.toList())
).get(5, TimeUnit.SECONDS);
} catch (InterruptedException | ExecutionException | TimeoutException e) {
e.printStackTrace();
//separate out the TimeoutException if you want to handle it differently
}
System.out.println(collect); //would be null in case of any exception
you can try CompletableFuture's overloaded supplyAsync method with executor parameter (CompletableFuture.supplyAsync(getStringSupplier(x), timeoutExecutorService)) and can refer link for timeoutExecutorService.
Related
How to print elements in buffer when using backpressure:
Flowable.range(1, 1000)
.onBackpressureBuffer(20, () ->{}, BackpressureOverflowStrategy.DROP_OLDEST)
.onBackpressureDrop(v -> System.out.println("Dropped.. :" + v))
.delay(0, TimeUnit.MILLISECONDS, Schedulers.io())
.doOnNext(value -> System.out.println("got " + value))
.map(value -> {
try {
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
return value;
})
.observeOn(Schedulers.newThread(),false, 20)
.subscribe(
value -> System.out.println("handled: " + value),
Throwable::printStackTrace,
() -> System.out.println("completed")
);
sleep(10000);
the output:
got 1
Dropped.. :21
Dropped.. :22
Dropped.. :23
Dropped.. :24
Dropped.. :25
.
.
.
I want
got 1
elements in buffer: 2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
Dropped.. :21
.
.
.
I do not know if that is possible or not.
Please help I'am new in reactive programming
Thanks
How can I recover from an exception thrown in the Sink of Akka Streams?
Simple Example:
Source<Integer, NotUsed> integerSource = Source.from(Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9));
integerSource.runWith(Sink.foreach(x -> {
if (x == 4) {
throw new Exception("Error Occurred");
}
System.out.println("Sink: " + x);
}), system);
Output:
Sink: 1
Sink: 2
Sink: 3
How can I handle the exception and move on to the next element from the source? (aka 5,6,7,8,9)
By default, the supervision strategy stops the stream when an exception is thrown. To change the supervision strategy to drop an exception-causing message and proceed to the next message, use the "resume" strategy. For example:
final Function<Throwable, Supervision.Directive> decider =
exc -> {
return Supervision.resume();
};
final Sink<Integer, CompletionStage<Done>> printSink =
Sink.foreach(x -> {
if (x == 4) {
throw new Exception("Error Occurred");
}
System.out.println("Sink: " + x);
});
final RunnableGraph<CompletionStage<Done>> runnableGraph =
integerSource.toMat(printSink, Keep.right());
final RunnableGraph<CompletionStage<Done>> withResumingSupervision =
runnableGraph.withAttributes(ActorAttributes.withSupervisionStrategy(decider));
final CompletionStage<Done> result = withResumingSupervision.run(system);
You could also define different supervision strategies for different kinds of exceptions:
final Function<Throwable, Supervision.Directive> decider =
exc -> {
if (exc instanceof MySpecificException) return Supervision.resume();
else return Supervision.stop();
};
I have a list of entities of type T. I also have a functional interface which acts as Supplier which has the method to performTask on entity and send back the result R which looks like:
R performTask(T entity) throws Exception.
I want to filter both: the successful results and errors & exceptions coming out of it onto separate maps. The code I wrote here is taking time, Kindly suggest what can be done.
I am looping on the list of entities, then process their completable future one by one, which I think is not the right way to do. Can you all suggest what can be done here ?
private void updateResultAndExceptionMaps(List < T > entities, final TaskProcessor < T, R > taskProcessor) {
ExecutorService executor = createExecutorService();
Map < T, R > outputMap = Collections.synchronizedMap(new HashMap < T, R > ());
Map < T, Exception > errorMap = new ConcurrentHashMap < T, Exception > ();
try {
entities.stream()
.forEach(entity -> CompletableFuture.supplyAsync(() -> {
try {
return taskProcessor.performTask(entity);
} catch (Exception e) {
errorMap.put(entity, (Exception) e.getCause());
LOG.error("Error processing entity Exception: " + entity, e);
}
return null;
}, executor)
.exceptionally(throwable -> {
errorMap.put(entity, (Exception) throwable);
LOG.error("Error processing entity Throwable: " + entity, throwable);
return null;
})
.thenAcceptAsync(R -> outputMap.put(entity, R))
.join()
); // end of for-each
LOG.info("outputMap Map -> " + outputMap);
LOG.info("errorMap Map -> " + errorMap);
} catch (Exception ex) {
LOG.warn("Error: " + ex, ex);
} finally {
executor.shutdown();
}
}
outputmap should contain the entity and result, R.
errorMap should contain entity and Exception.
This is because you iterate over List of entities one by one, create CompletableFuture object and immediately block iteration because of join method which waits until given processor finishes it work or throw exception. You can do that with full multithreading support by converting each entity to CompletableFuture, collect all CompletableFuture instances and after that wait for all invoking join on each.
Below code should do the trick in your case:
entities.stream()
.map(entity -> CompletableFuture.supplyAsync(() -> {
try {
return taskProcessor.performTask(entity);
} catch (Exception e) {
errorMap.put(entity, (Exception) e.getCause());
}
return null;
}, executor)
.exceptionally(throwable -> {
errorMap.put(entity, (Exception) throwable);
return null;
})
.thenAcceptAsync(R -> outputMap.put(entity, R))
).collect(Collectors.toList())
.forEach(CompletableFuture::join);
I have 2 URLs to fetch the data, for example: \location_1\{userid} and \location_2\{userid}. for the first i get the list of the users and then need to fetch user details by above requests. the issue is that i need to call the \location_1\{userid} and in case there is an error(exception) fetch the data from \location_2\{userid}. is it possible to make it with single rx-chain? i've tried try/catch as described here but looks catch newer calls, only onErrorResumeNext calls.
Observable<List<TestModel2>> observable = apiTest
.performTest()
.flatMapIterable(items -> items)
.flatMap(testModel -> {
try
{
return apiTest.performTest2(testModel.userId);
} catch (Exception e)
{
return apiTest.performTest3(testModel.userId);
}
}).doOnNext(testModel2 -> {Log.d("TestItemData", "doOnNext --- " + testModel2.title);})
.onErrorResumeNext(throwable ->{
Log.d("TestItemData", "onErrorResumeNext -------- ");
return Observable.empty();
})
.toList()
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread());
Use onErrorResumeNext (as you already did a bit later in the flow):
Observable<List<TestModel2>> observable = apiTest
.performTest()
.flatMapIterable(items -> items)
.flatMap(testModel ->
apiTest.performTest2(testModel.userId)
.onErrorResumeNext(e -> apiTest.performTest3(testModel.userId)); // <----------------
)
.doOnNext(testModel2 -> {
Log.d("TestItemData", "doOnNext --- " + testModel2.title);
})
.onErrorResumeNext(throwable ->{
Log.d("TestItemData", "onErrorResumeNext -------- ");
return Observable.empty();
})
.toList()
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread());
I'm looking for a way to attach multiple subscribers to an RxJava Observable stream, with each subscriber processing emitted events asynchronously.
I first tried using .flatMap() but that didn't seem to work on any subsequent subscribers. All subscribers were processing events on the same thread.
.flatMap(s -> Observable.just(s).subscribeOn(Schedulers.newThread()))
What ended up working was consuming each event in a new thread by creating a new Observable each time:
Observable.from(Arrays.asList(new String[]{"1", "2", "3"}))
.subscribe(j -> {
Observable.just(j)
.subscribeOn(Schedulers.newThread())
.subscribe(i -> {
try {
Thread.sleep(ThreadLocalRandom.current().nextInt(100, 500));
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("s1=>" + Thread.currentThread().getName() + "=>" + i);
});
});
Output:
s1=>RxNewThreadScheduler-1=>1
s1=>RxNewThreadScheduler-2=>2
s1=>RxNewThreadScheduler-3=>3
And the end result with multiple subscribers:
ConnectableObservable<String> e = Observable.from(Arrays.asList(new String[]{"1", "2", "3"}))
.publish();
e.subscribe(j -> {
Observable.just(j)
.subscribeOn(Schedulers.newThread())
.subscribe(i -> {
try {
Thread.sleep(ThreadLocalRandom.current().nextInt(100, 500));
} catch (InterruptedException e1) {
e1.printStackTrace();
}
System.out.println("s1=>" + Thread.currentThread().getName() + "=>" + i);
});
});
e.subscribe(j -> {
Observable.just(j)
.subscribeOn(Schedulers.newThread())
.subscribe(i -> {
try {
Thread.sleep(ThreadLocalRandom.current().nextInt(100, 500));
} catch (InterruptedException e1) {
e1.printStackTrace();
}
System.out.println("s2=>" + Thread.currentThread().getName() + "=>" + i);
});
});
e.connect();
Output:
s2=>RxNewThreadScheduler-4=>2
s1=>RxNewThreadScheduler-1=>1
s1=>RxNewThreadScheduler-3=>2
s2=>RxNewThreadScheduler-6=>3
s2=>RxNewThreadScheduler-2=>1
s1=>RxNewThreadScheduler-5=>3
However, this seems a little clunky. Is there a more elegant solution or is RxJava just not a good use case for this?
Use .flatMap(s -> Observable.just(s).observeOn(Schedulers.newThread())....)
if I understood the rx-contract correctly, you are trying to do something, which is against it.
Lets have a look at the contract
The contract of an RxJava Observable is that events ( onNext() , onCompleted() , onEr
ror() ) can never be emitted concurrently. In other words, a single Observable
stream must always be serialized and thread-safe. Each event can be emitted from a
different thread, as long as the emissions are not concurrent. This means no inter‐
leaving or simultaneous execution of onNext() . If onNext() is still being executed on
one thread, another thread cannot begin invoking it again (interleaving). --Tomasz Nurkiewicz in Reactive Programming with RxJava
In my opinion you are trying to break the contract by using a nested subscription in the outer subscription. The onNext call to the subscriber is not serialized anymore.
Why not move the "async"-workload from the subscriber to a flatMap-operator and subscribe to the new observable:
ConnectableObservable<String> stringObservable = Observable.from(Arrays.asList(new String[]{"1", "2", "3"}))
.flatMap(s -> {
return Observable.just(s).subscribeOn(Schedulers.computation());
})
.publish();
stringObservable
.flatMap(s -> {
// do More asyncStuff depending on subscription
return Observable.just(s).subscribeOn(Schedulers.newThread());
})
.subscribe(s -> {
// use result here
});
stringObservable
.subscribe(s -> {
// use immediate result here.
});
stringObservable.connect();
flatMap along with doOnNext on the Observable inside the flatMap will result in the same output as yours.
onNext() is always called in a sequential manner hence using doOnNext after the flatMap will also not work for you. Due to the same reason writing the action inside the final subscribe didn't work in your case.
The below code is written using RxJava2. In version 1 of RxJava you will have to add the try-catch block around Thread.sleep.
ConnectableObservable<String> e = Observable.just("1", "2", "3").publish();
e.flatMap(
s -> Observable.just(s)
.subscribeOn(Schedulers.newThread())
.doOnNext(i -> { // <<<<<<
Thread.sleep(ThreadLocalRandom.current().nextInt(100, 500));
System.out.println("s1=>" + Thread.currentThread().getName() + "=>" + i);
}))
.subscribe();
e.flatMap(
s -> Observable.just(s)
.subscribeOn(Schedulers.newThread())
.doOnNext(i -> { // <<<<<<
Thread.sleep(ThreadLocalRandom.current().nextInt(100, 500));
System.out.println("s2=>" + Thread.currentThread().getName() + "=>" + i);
}))
.subscribe();
e.connect();
You can achieve it with Flowable and parallel:
Flowable.fromIterable(Arrays.asList("1", "2", "3"))
.parallel(3)
.runOn(Schedulers.newThread())
.map(item -> {
try {
Thread.sleep(ThreadLocalRandom.current().nextInt(100, 500));
} catch (InterruptedException e1) {
e1.printStackTrace();
}
System.out.println("s1=>" + Thread.currentThread().getName() + "=>" + item);
return Completable.complete();
})
.sequential().subscribe();