I have an async API that essentially returns results through pagination
public CompletableFuture<Response> getNext(int startFrom);
Each Response object contains a list of offsets from startFrom and a flag indicating whether there are more elements remaining and, therefore, another getNext() request to make.
I'd like to write a method that goes through all the pages and retrieves all the offsets. I can write it in a synchronous manner like so
int startFrom = 0;
List<Integer> offsets = new ArrayList<>();
for (;;) {
CompletableFuture<Response> future = getNext(startFrom);
Response response = future.get(); // an exception stops everything
if (response.getOffsets().isEmpty()) {
break; // we're done
}
offsets.addAll(response.getOffsets());
if (!response.hasMore()) {
break; // we're done
}
startFrom = getLast(response.getOffsets());
}
In other words, we call getNext() with startFrom at 0. If an exception is thrown, we short-circuit the entire process. Otherwise, if there are no offsets, we complete. If there are offsets, we add them to the master list. If there are no more left to fetch, we complete. Otherwise, we reset the startFrom to the last offset we fetched and repeat.
Ideally, I want to do this without blocking with CompletableFuture::get() and returning a CompletableFuture<List<Integer>> containing all the offsets.
How can I do this? How can I compose the futures to collect their results?
I'm thinking of a "recursive" (not actually in execution, but in code)
private CompletableFuture<List<Integer>> recur(int startFrom, List<Integer> offsets) {
CompletableFuture<Response> future = getNext(startFrom);
return future.thenCompose((response) -> {
if (response.getOffsets().isEmpty()) {
return CompletableFuture.completedFuture(offsets);
}
offsets.addAll(response.getOffsets());
if (!response.hasMore()) {
return CompletableFuture.completedFuture(offsets);
}
return recur(getLast(response.getOffsets()), offsets);
});
}
public CompletableFuture<List<Integer>> getAll() {
List<Integer> offsets = new ArrayList<>();
return recur(0, offsets);
}
I don't love this, from a complexity point of view. Can we do better?
I also wanted to give a shot at EA Async on this one, as it implements Java support for async/await (inspired from C#). So I just took your initial code, and converted it:
public CompletableFuture<List<Integer>> getAllEaAsync() {
int startFrom = 0;
List<Integer> offsets = new ArrayList<>();
for (;;) {
// this is the only thing I changed!
Response response = Async.await(getNext(startFrom));
if (response.getOffsets().isEmpty()) {
break; // we're done
}
offsets.addAll(response.getOffsets());
if (!response.hasMore()) {
break; // we're done
}
startFrom = getLast(response.getOffsets());
}
// well, you also have to wrap your result in a future to make it compilable
return CompletableFuture.completedFuture(offsets);
}
You then have to instrument your code, for example by adding
Async.init();
at the beginning of your main() method.
I must say: this really looks like magic!
Behind the scenes, EA Async notices there is an Async.await() call within the method, and rewrites it to handle all the thenCompose()/thenApply()/recursion for you. The only requirement is that your method must return a CompletionStage or CompletableFuture.
That's really async code made easy!
For the exercise, I made a generic version of this algorithm, but it is rather complex because you need:
an initial value to call the service (the startFrom)
the service call itself (getNext())
a result container to accumulate the intermediate values (the offsets)
an accumulator (offsets.addAll(response.getOffsets()))
a condition to perform the "recursion" (response.hasMore())
a function to compute the next input (getLast(response.getOffsets()))
so this gives:
public <T, I, R> CompletableFuture<R> recur(T initialInput, R resultContainer,
Function<T, CompletableFuture<I>> service,
BiConsumer<R, I> accumulator,
Predicate<I> continueRecursion,
Function<I, T> nextInput) {
return service.apply(initialInput)
.thenCompose(response -> {
accumulator.accept(resultContainer, response);
if (continueRecursion.test(response)) {
return recur(nextInput.apply(response),
resultContainer, service, accumulator,
continueRecursion, nextInput);
} else {
return CompletableFuture.completedFuture(resultContainer);
}
});
}
public CompletableFuture<List<Integer>> getAll() {
return recur(0, new ArrayList<>(), this::getNext,
(list, response) -> list.addAll(response.getOffsets()),
Response::hasMore,
r -> getLast(r.getOffsets()));
}
A small simplification of recur() is possible by replacing initialInput by the CompletableFuture returned by the result of the first call, the resultContainer and the accumulator can be merged into a single Consumer and the service can then be merged with the nextInput function.
But this gives a little more complex getAll():
private <I> CompletableFuture<Void> recur(CompletableFuture<I> future,
Consumer<I> accumulator,
Predicate<I> continueRecursion,
Function<I, CompletableFuture<I>> service) {
return future.thenCompose(result -> {
accumulator.accept(result);
if (continueRecursion.test(result)) {
return recur(service.apply(result), accumulator, continueRecursion, service);
} else {
return CompletableFuture.completedFuture(null);
}
});
}
public CompletableFuture<List<Integer>> getAll() {
ArrayList<Integer> resultContainer = new ArrayList<>();
return recur(getNext(0),
result -> resultContainer.addAll(result.getOffsets()),
Response::hasMore,
r -> getNext(getLast(r.getOffsets())))
.thenApply(unused -> resultContainer);
}
Related
I have these two methods which call an async API and return a Mono<Boolean> if a value exists. I am returning a random boolean value for the sake of this example,
private Mono<Boolean> checkFirstExists() {
// Replacing actual API call here
return Mono.just(Boolean.FALSE);
}
private Mono<Boolean> checkSecondExists() {
// Replacing actual API call here
return Mono.just(Boolean.TRUE);
}
Now, I have another method that should combine the results of these two methods and simply return a boolean if either checkFirstExists or checkSecondExists is true.
private boolean checkIfExists() {
// Should return true if any of the underlying method returns true
final Flux<Boolean> exists = Flux.concat(checkFirstExists(), checkSecondExists());
return exists.blockFirst();
}
What's the best way of doing this? Mono.zip maybe? Any help would be great.
Mono.zip is the correct approach for awaiting completion of multiple async operations before continuing. Something like this should work:
return Mono.zip(checkFirstExists(), checkSecondExists(), (first, second) -> first && second);
Or if a list is provided instead:
private boolean checkIfExists()
{
return allTrue(Arrays.asList(checkFirstExists(), checkSecondExists())).blockOptional().orElseThrow(() -> new IllegalStateException("Invalid State"));
}
private Mono<Boolean> allTrue(List<Mono<Boolean>> toAggregate)
{
return mergeMonos(toAggregate).map(list -> list.stream().allMatch(val -> val));
}
#SuppressWarnings("unchecked")
private <T> Mono<List<T>> mergeMonos(List<Mono<T>> toAggregate)
{
return Mono.zip(toAggregate, array -> Stream.of(array).map(o -> (T) o).collect(Collectors.toList()));
}
Unrelated Note:
In general, it is worth keeping the operation async as long as possible when constructing reactive flows. It may be worth having the 'checkIfExists' function return a Mono instead of blocking.
I have this:
Stream<CompletableFuture<List<Item>>>
how can I convert it to
Stream<CompletableFuture<Item>>
Where: the second stream is comprised of each and all the Items inside each of the lists in the first stream.
I looked into thenCompose but that solves a completely different problem which is also referred to as "flattening".
How can this be done efficiently, in a streaming fashion, without blocking or prematurely consuming more stream items than necessary?
Here is my best attempt so far:
ExecutorService pool = Executors.newFixedThreadPool(PARALLELISM);
Stream<CompletableFuture<List<IncomingItem>>> reload = ... ;
#SuppressWarnings("unchecked")
CompletableFuture<List<IncomingItem>> allFutures[] = reload.toArray(CompletableFuture[]::new);
CompletionService<List<IncomingItem>> queue = new ExecutorCompletionService<>(pool);
for(CompletableFuture<List<IncomingItem>> item: allFutures) {
queue.submit(item::get);
}
List<IncomingItem> THE_END = new ArrayList<IncomingItem>();
CompletableFuture<List<IncomingItem>> ender = CompletableFuture.allOf(allFutures).thenApply(whatever -> {
queue.submit(() -> THE_END);
return THE_END;
});
queue.submit(() -> ender.get());
Iterable<List<IncomingItem>> iter = () -> new Iterator<List<IncomingItem>>() {
boolean checkNext = true;
List<IncomingItem> next = null;
#Override
public boolean hasNext() {
if(checkNext) {
try {
next = queue.take().get();
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
}
checkNext = false;
}
if(next == THE_END || next == null) {
return false;
}
else {
return true;
}
}
#Override
public List<IncomingItem> next() {
if(checkNext) {
hasNext();
}
if(!hasNext()) {
throw new IllegalStateException();
}
checkNext = true;
return next;
}
};
Stream<IncomingItem> flat = StreamSupport.stream(iter.spliterator(), false).flatMap(List::stream);
This works at first, unfortunately, it has a fatal bug: the resulting stream seems to terminate prematurely, before retrieving all the items.
As I wrote in my comment, this is impossible.
Consider a some arbitrary service, which will return a CompletableFuture<Integer>:
CompletableFuture<Integer> getDiceRoll();
I can now convert this CompletableFuture<Integer> to a Stream<CompletableFuture<List<Object>>> without any problem:
Stream<CompletableFuture<List<Object>>> futureList = Stream.of(getDiceRoll().thenApply(n -> List.of(new Object[n])));
Let's suppose there would be a general way to turn a Stream<CompletableFuture<List<T>>> into a Stream<CompletableFuture<T>>:
<T> Stream<CompletableFuture<T> magic(Stream<CompletableFuture<List<T>>> arg);
Then I can do the following:
int diceRoll = magic(Stream.of(getDiceRoll().thenApply(n -> List.of(new Object[n])))).count();
Wait, what?
I am now able to get an arbitrary integer out of a CompletableFuture.
Which means, with some engineering effort I can get all the information out of a CompletableFuture - after all, memory is just some numbers.
So we have to conclude that a method like magic can not exist, without violating the time fabric.
And this is the answer: There is no such method, because it can not exist.
Agreed with Johannes Kuhn. You can't know Futures's state while it's still executing and thus can not convert from Stream<CompletableFuture<List>>
to Stream<CompletableFuture> .
Although the output of stream can be merged using following piece of code -
java Stream<CompletableFuture<List<Item>>> to java List<Item> or
java List<CompletableFuture<List<AuditRecord>>> to java List<Item>
List<Item> output = input.map(CompletableFuture::join).collect(toList()).stream()
.flatMap(Collection::stream).collect(toList());
With Java 8, I have this code:
if(element.exist()){
// Do something
}
I want to convert to lambda style,
element.ifExist(el -> {
// Do something
});
with an ifExist method like this:
public void ifExist(Consumer<Element> consumer) {
if (exist()) {
consumer.accept(this);
}
}
But now I have else cases to call:
element.ifExist(el -> {
// Do something
}).ifNotExist(el -> {
// Do something
});
I can write a similar ifNotExist, and I want they are mutually exclusive (if the exist condition is true, there is no need to check ifNotExist, because sometimes, the exist() method takes so much workload to check), but I always have to check two times. How can I avoid that?
Maybe the "exist" word make someone misunderstand my idea. You can imagine that I also need some methods:
ifVisible()
ifEmpty()
ifHasAttribute()
Many people said that this is bad idea, but:
In Java 8 we can use lambda forEach instead of a traditional for loop. In programming for and if are two basic flow controls. If we can use lambda for a for loop, why is using lambda for if bad idea?
for (Element element : list) {
element.doSomething();
}
list.forEach(Element::doSomething);
In Java 8, there's Optional with ifPresent, similar to my idea of ifExist:
Optional<Elem> element = ...
element.ifPresent(el -> System.out.println("Present " + el);
And about code maintenance and readability, what do you think if I have the following code with many repeating simple if clauses?
if (e0.exist()) {
e0.actionA();
} else {
e0.actionB();
}
if (e1.exist()) {
e0.actionC();
}
if (e2.exist()) {
e2.actionD();
}
if (e3.exist()) {
e3.actionB();
}
Compare to:
e0.ifExist(Element::actionA).ifNotExist(Element::actionB);
e1.ifExist(Element::actionC);
e2.ifExist(Element::actionD);
e3.ifExist(Element::actionB);
Which is better? And, oops, do you notice that in the traditional if clause code, there's a mistake in:
if (e1.exist()) {
e0.actionC(); // Actually e1
}
I think if we use lambda, we can avoid this mistake!
As it almost but not really matches Optional, maybe you might reconsider the logic:
Java 8 has a limited expressiveness:
Optional<Elem> element = ...
element.ifPresent(el -> System.out.println("Present " + el);
System.out.println(element.orElse(DEFAULT_ELEM));
Here the map might restrict the view on the element:
element.map(el -> el.mySpecialView()).ifPresent(System.out::println);
Java 9:
element.ifPresentOrElse(el -> System.out.println("Present " + el,
() -> System.out.println("Not present"));
In general the two branches are asymmetric.
It's called a 'fluent interface'. Simply change the return type and return this; to allow you to chain the methods:
public MyClass ifExist(Consumer<Element> consumer) {
if (exist()) {
consumer.accept(this);
}
return this;
}
public MyClass ifNotExist(Consumer<Element> consumer) {
if (!exist()) {
consumer.accept(this);
}
return this;
}
You could get a bit fancier and return an intermediate type:
interface Else<T>
{
public void otherwise(Consumer<T> consumer); // 'else' is a keyword
}
class DefaultElse<T> implements Else<T>
{
private final T item;
DefaultElse(final T item) { this.item = item; }
public void otherwise(Consumer<T> consumer)
{
consumer.accept(item);
}
}
class NoopElse<T> implements Else<T>
{
public void otherwise(Consumer<T> consumer) { }
}
public Else<MyClass> ifExist(Consumer<Element> consumer) {
if (exist()) {
consumer.accept(this);
return new NoopElse<>();
}
return new DefaultElse<>(this);
}
Sample usage:
element.ifExist(el -> {
//do something
})
.otherwise(el -> {
//do something else
});
You can use a single method that takes two consumers:
public void ifExistOrElse(Consumer<Element> ifExist, Consumer<Element> orElse) {
if (exist()) {
ifExist.accept(this);
} else {
orElse.accept(this);
}
}
Then call it with:
element.ifExistOrElse(
el -> {
// Do something
},
el -> {
// Do something else
});
The problem
(1) You seem to mix up different aspects - control flow and domain logic.
element.ifExist(() -> { ... }).otherElementMethod();
^ ^
control flow method business logic method
(2) It is unclear how methods after a control flow method (like ifExist, ifNotExist) should behave. Should they be always executed or be called only under the condition (similar to ifExist)?
(3) The name ifExist implies a terminal operation, so there is nothing to return - void. A good example is void ifPresent(Consumer) from Optional.
The solution
I would write a fully separated class that would be independent of any concrete class and any specific condition.
The interface is simple, and consists of two contextless control flow methods - ifTrue and ifFalse.
There can be a few ways to create a Condition object. I wrote a static factory method for your instance (e.g. element) and condition (e.g. Element::exist).
public class Condition<E> {
private final Predicate<E> condition;
private final E operand;
private Boolean result;
private Condition(E operand, Predicate<E> condition) {
this.condition = condition;
this.operand = operand;
}
public static <E> Condition<E> of(E element, Predicate<E> condition) {
return new Condition<>(element, condition);
}
public Condition<E> ifTrue(Consumer<E> consumer) {
if (result == null)
result = condition.test(operand);
if (result)
consumer.accept(operand);
return this;
}
public Condition<E> ifFalse(Consumer<E> consumer) {
if (result == null)
result = condition.test(operand);
if (!result)
consumer.accept(operand);
return this;
}
public E getOperand() {
return operand;
}
}
Moreover, we can integrate Condition into Element:
class Element {
...
public Condition<Element> formCondition(Predicate<Element> condition) {
return Condition.of(this, condition);
}
}
The pattern I am promoting is:
work with an Element;
obtain a Condition;
control the flow by the Condition;
switch back to the Element;
continue working with the Element.
The result
Obtaining a Condition by Condition.of:
Element element = new Element();
Condition.of(element, Element::exist)
.ifTrue(e -> { ... })
.ifFalse(e -> { ... })
.getOperand()
.otherElementMethod();
Obtaining a Condition by Element#formCondition:
Element element = new Element();
element.formCondition(Element::exist)
.ifTrue(e -> { ... })
.ifFalse(e -> { ... })
.getOperand()
.otherElementMethod();
Update 1:
For other test methods, the idea remains the same.
Element element = new Element();
element.formCondition(Element::isVisible);
element.formCondition(Element::isEmpty);
element.formCondition(e -> e.hasAttribute(ATTRIBUTE));
Update 2:
It is a good reason to rethink the code design. Neither of 2 snippets is great.
Imagine you need actionC within e0.exist(). How would the method reference Element::actionA be changed?
It would be turned back into a lambda:
e0.ifExist(e -> { e.actionA(); e.actionC(); });
unless you wrap actionA and actionC in a single method (which sounds awful):
e0.ifExist(Element::actionAAndC);
The lambda now is even less 'readable' then the if was.
e0.ifExist(e -> {
e0.actionA();
e0.actionC();
});
But how much effort would we make to do that? And how much effort will we put into maintaining it all?
if(e0.exist()) {
e0.actionA();
e0.actionC();
}
If you are performing a simple check on an object and then executing some statements based on the condition then one approach would be to have a Map with a Predicate as key and desired expression as value
for example.
Map<Predicate<Integer>,Supplier<String>> ruleMap = new LinkedHashMap <Predicate<Integer>,Supplier<String>>(){{
put((i)-> i<10,()->"Less than 10!");
put((i)-> i<100,()->"Less than 100!");
put((i)-> i<1000,()->"Less than 1000!");
}};
We could later stream the following Map to get the value when the Predicate returns true which could replace all the if/else code
ruleMap.keySet()
.stream()
.filter((keyCondition)->keyCondition.test(numItems,version))
.findFirst()
.ifPresent((e)-> System.out.print(ruleMap.get(e).get()));
Since we are using findFirst() it is equivalent to if/else if /else if ......
I want my code to repeat a certain asynchronous operation until this operation is successful (i.e. until it returns true).
At the moment I'm using the following workaround:
Supplier<Observable<Boolean>> myOperation = () -> {
// do something useful and return 'true' if it was successful
// NOTE: GENERATING A RANDOM NUMBER IS JUST AN EXAMPLE HERE
// I WANT TO RUN AN ASYNCHRONOUS OPERATION (LIKE PINGING A SERVER
// OR THE LIKE) AND RETRY IT UNTIL IT SUCCEEDS.
System.out.println("Try");
return Observable.just(Math.random() > 0.9);
};
final Throwable retry = new IllegalStateException();
Observable.<Boolean>create(subscriber -> {
myOperation.get().subscribe(subscriber);
}).flatMap(b -> b ? Observable.just(b) : Observable.error(retry))
.retryWhen(exceptions -> exceptions.flatMap(exception -> {
if (exception == retry) {
return Observable.timer(1, TimeUnit.SECONDS);
}
return Observable.error(exception);
}))
.toBlocking()
.forEach(b -> {
System.out.println("Connected.");
});
It works well and prints out something like this:
Try
Try
...
Try
Connected.
The code does what I want, but it doesn't look very elegant. I'm sure there must be a better way. Maybe by using a custom Operator?
Does anybody know how to achieve the same thing in RxJava but in a more readable manner and without the artificial Throwable?
Not enough time, so this is going to be by memory...
public class Randomizer implements Iterable<Double>, Iterator<Double> {
public Iterator<Double> getIterator() {return this;}
public boolean hasNext() {return true;}
public Double next() {return Math.random();}
}
...
Observable.from(new Randomizer())
.takeWhile(value -> value < 0.99);
// or takeUntil(value -> value > 0.99); can't remember their differences.
OTOH if you need to do something more complex, look into Observable.defer() and / or a BehaviorSubject.
Edit: Now there's a bit more time to read your post, you could try something like this:
Observable.defer(() -> createConnectionObservable())
.retry((count, err) -> {
if(count>9) return false;
if(!(err instanceof IOException)) return false;
return true;
})
Keep in mind that if you use Retrofit you shouldn't need defer(), as retrofit will re-initiate the call when a new subscription happens.
How can I express this with java8 streaming-API?
I want to perform itemConsumer for every item of a stream. If there
are no items I want to perform emptyAction.
Of course I could write something like this:
Consumer<Object> itemConsumer = System.out::println;
Runnable emptyAction = () -> {System.out.println("no elements");};
Stream<Object> stream = Stream.of("a","b"); // or Stream.empty()
List<Object> list = stream.collect(Collectors.toList());
if (list.isEmpty())
emptyAction.run();
else
list.stream().forEach(itemConsumer);
But I would prefer to avoid any Lists.
I also thought about setting a flag in a peek method - but that flag would be non-final and therefore not allowed. Using a boolean container also seems to be too much of a workaround.
You could coerce reduce to do this. The logic would be to reduce on false, setting the value to true if any useful data is encountered.
The the result of the reduce is then false then no items have been encountered. If any items were encountered then the result would be true:
boolean hasItems = stream.reduce(false, (o, i) -> {
itemConsumer.accept(i);
return true;
}, (l, r) -> l | r);
if (!hasItems) {
emptyAction.run();
}
This should work fine for parallel streams, as any stream encountering an item would set the value to true.
I'm not sure, however, that I like this as it's a slightly obtuse use of the reduce operation.
An alternative would be to use AtomicBoolean as a mutable boolean container:
final AtomicBoolean hasItems = new AtomicBoolean(false);
stream.forEach(i -> {
itemConsumer.accept(i);
hasItems.set(true);
});
if (!hasItems.get()) {
emptyAction.run();
}
I don't know if I like that more or less however.
Finally, you could have your itemConsumer remember state:
class ItemConsumer implements Consumer<Object> {
private volatile boolean hasConsumedAny;
#Override
public void accept(Object o) {
hasConsumedAny = true;
//magic magic
}
public boolean isHasConsumedAny() {
return hasConsumedAny;
}
}
final ItemConsumer itemConsumer = new ItemConsumer();
stream.forEach(itemConsumer::accept);
if (!itemConsumer.isHasConsumedAny()) {
emptyAction.run();
}
This seems a bit neater, but might not be practical. So maybe a decorator pattern -
class ItemConsumer<T> implements Consumer<T> {
private volatile boolean hasConsumedAny;
private final Consumer<T> delegate;
ItemConsumer(final Consumer<T> delegate) {
this.delegate = delegate;
}
#Override
public void accept(T t) {
hasConsumedAny = true;
delegate.accept(t);
}
public boolean isHasConsumedAny() {
return hasConsumedAny;
}
}
final ItemConsumer<Object> consumer = new ItemConsumer<Object>(() -> /** magic **/);
TL;DR: something has to remember whether you encountered anything during the consumption of the Stream, be it:
the Stream itself in case of reduce;
AtomicBoolean; or
the consumer
I think the consumer is probably best placed, from a logic point of view.
A solution without any additional variables:
stream.peek(itemConsumer).reduce((a, b) -> a).orElseGet(() -> {
emptyAction.run();
return null;
});
Note that if the stream is parallel, then itemConsumer could be called simultaneously for different elements in different threads (like in forEach, not in forEachOrdered). Also this solution will fail if the first stream element is null.
There’s a simple straight-forward solution:
Spliterator<Object> sp=stream.spliterator();
if(!sp.tryAdvance(itemConsumer))
emptyAction.run();
else
sp.forEachRemaining(itemConsumer);
You can even keep parallel support for the elements after the first, if you wish:
Spliterator<Object> sp=stream.parallel().spliterator();
if(!sp.tryAdvance(itemConsumer))
emptyAction.run();
else
StreamSupport.stream(sp, true).forEach(itemConsumer);
In my opinion, it is much easier to understand as a reduce based solution.
You could do this:
if(stream.peek(itemConsumer).count() == 0){
emptyAction.run();
}
But it seems that count may be changed to skip the peek if it knows the size of the Stream in Java 9 (see here), so if you want it to work in the future you could use:
if(stream.peek(itemConsumer).mapToLong(e -> 1).sum() == 0){
emptyAction.run();
}
Another attempt to use reduce:
Stream<Object> stream = Stream.of("a","b","c");
//Stream<Object> stream = Stream.empty();
Runnable defaultRunnable = () -> System.out.println("empty Stream");
Consumer<Object> printConsumer = System.out::println;
Runnable runnable = stream.map(x -> toRunnable(x, printConsumer)).reduce((a, b) -> () -> {
a.run();
b.run();
}).orElse(defaultRunnable);
runnable.run(); // prints a, b, c (or empty stream when it is empty)
// for type inference
static <T> Runnable toRunnable(T t, Consumer<T> cons){
return ()->cons.accept(t);
}
This approach does not use peek() which according to Javadoc "mainly exists to support debugging"