Develop Reference

Webflux producer consumer problem (webClient)

Hi I have problem with WebFlux and backpressure:
Flux.range(0, 100)
.flatMap((Integer y) -> {
return reallySlowApi();
})
.doOnEach((Signal<String> x1) -> {
log("next-------" );
})
.subscribeOn(Schedulers.elastic())
.subscribe()
;
How I can limit calls like to one call per 5 seconds. Note: only reallySlowApi can be modified.
private Mono<String> reallySlowApi() {
return webClient
.get()
.retrieve()
.bodyToMono(String.class);
}
Edit: I know about delayElements but it won't resolve issue if Api will get even slower. I need optimal way of working with reallySlowApi.

One way is with delayElements()
public void run() {
Flux.range(0, 100)
.delayElements(Duration.ofSeconds(5)) // only emit every 5 seconds
.flatMap(y -> reallySlowApi())
.doOnNext(x1 -> System.out.println("next-------"))
.blockLast(); // subscribe AND wait for the flux to complete
}
private Mono<String> reallySlowApi() {
return Mono.just("next");
}
You could also use Flux.interval() plus a take() to limit the number of iterations.
Flux.interval(Duration.ofSeconds(5))
.take(100)
Note that the subscribeOn in your example doesn't do anything partcularly as the subscribe operation applies to the generation of the range 0-100 which is not blocking.

You can use retry mechanisam in ur webclient code
.doOnError(error -> handleError(error.getMessage()))
.timeout(Duration.ofSeconds(ServiceConstants.FIVE))
.retryWhen(
Retry.backoff(retryCount, Duration.ofSeconds(ServiceConstants.FIVE))
.filter(throwable -> throwable instanceof TimeoutException)
)

Just to put here solution that I found. WebFlux when mapping response we can pass concurrency parameter that solve this issue.
flatMap(mapper, concurrency)
.flatMap((Integer y) -> {
return reallySlowApi();
} , 3)

RxJava retryWhen (exponential back-off) not working

So I know this has been asked many times before, but I have tried many things and nothing seems to work.
Let's start with these blogs/articles/code:
https://blog.danlew.net/2016/01/25/rxjavas-repeatwhen-and-retrywhen-explained/
https://jimbaca.com/rxjava-retrywhen/
http://blog.inching.org/RxJava/2016-12-12-rx-java-error-handling.html
https://pamartinezandres.com/rxjava-2-exponential-backoff-retry-only-when-internet-is-available-5a46188ab175
https://gist.github.com/wotomas/35006d156a16345349a2e4c8e159e122
And many others.
In a nutshell all of them describe how you can use retryWhen to implement exponential back-off. Something like this:
source
.retryWhen(
errors -> {
return errors
.zipWith(Observable.range(1, 3), (n, i) -> i)
.flatMap(
retryCount -> {
System.out.println("retry count " + retryCount);
return Observable.timer((long) Math.pow(1, retryCount), SECONDS);
});
})
Even the documentation in the library agrees with it:
https://github.com/ReactiveX/RxJava/blob/3.x/src/main/java/io/reactivex/rxjava3/core/Observable.java#L11919.
However, I've tried this and some pretty similar variations, not worthy to describe here, and nothing seems to work. There's a way in that the examples works and is using blocking subscribers but I want to avoid blocking threads.
So if to the previous observable we apply a blocking subscriber like this:
.blockingForEach(System.out::println);
It works as expected. But as that's not the idea. If we try:
.subscribe(
x -> System.out.println("onNext: " + x),
Throwable::printStackTrace,
() -> System.out.println("onComplete"));
The flow runs only once, thus not what I want to achieve.
Does that mean it cannot be used as I'm trying to? From the documentation it doesn't seem to be a problem trying to accomplish my requirement.
Any idea what am I missing?
TIA.
Edit: There are 2 ways I'm testing this:
A test method (using testng):
Observable<Integer> source =
Observable.just("test")
.map(
x -> {
System.out.println("trying again");
return Integer.parseInt(x);
});
source
.retryWhen(
errors -> {
return errors
.zipWith(Observable.range(1, 3), (n, i) -> i)
.flatMap(
retryCount -> {
return Observable.timer((long) Math.pow(1, retryCount), SECONDS);
});
})
.subscribe(...);
From a Kafka consumer (using Spring boot):
This is only the subscription to the observer, but the retries logic is what I described earlier in the post.
#KafkaListener(topics = "${kafka.config.topic}")
public void receive(String payload) {
log.info("received payload='{}'", payload);
service
.updateMessage(payload)
.subscribe(...)
.dispose();
}

The main issue of your code is that Observable.timer is by default operating on the computation scheduler. This adds extra effort when trying to verify the behaviour within a test.
Here is some unit testing code that verifies that your retry code is actually retrying.
It adds a counter, just so we can easily check how many calls have happened.
It uses the TestScheduler instead of the computation scheduler so that we can pretend moving in time through advanceTimeBy.
TestScheduler testScheduler = new TestScheduler();
AtomicInteger counter = new AtomicInteger();
Observable<Integer> source =
Observable.just("test")
.map(
x -> {
System.out.println("trying again");
counter.getAndIncrement();
return Integer.parseInt(x);
});
TestObserver<Integer> testObserver = source
.retryWhen(
errors -> {
return errors
.zipWith(Observable.range(1, 3), (n, i) -> i)
.flatMap(
retryCount -> {
return Observable.timer((long) Math.pow(1, retryCount), SECONDS, testScheduler);
});
})
.test();
assertEquals(1, counter.get());
testScheduler.advanceTimeBy(1, SECONDS);
assertEquals(2, counter.get());
testScheduler.advanceTimeBy(1, SECONDS);
assertEquals(3, counter.get());
testScheduler.advanceTimeBy(1, SECONDS);
assertEquals(4, counter.get());
testObserver.assertComplete();

Is there an Observable which can make multiple 0 argument emissions?

I have an Observable which makes an emission every half second. When this Observable makes an emission, I do not care about the object which is emitted.
In this situation using a Completable is inadequate as a Completable can only make one zero argument emission.
This is what I am currently using, which works fine, but is imperfect
compositeDisposable.add(
Observable.interval(500L, TimeUnit.MILLISECONDS)
.timeInterval()
.observeOn(AndroidSchedulers.mainThread())
.flatMap(longTimed -> {
if (emissionBoolean) {
//todo: find an observable that can emit 0 arguments
return Observable.just(true);
}
return Observer::onComplete;
})
.subscribe(wishIWasAZeroArgumentBoolean -> {
onTick();
}));
this is what I want to have for my subscribe instead
.subscribe(() -> {
onTick();
}));

I think this is a valid question. Maybe cannot be used because Maybe does not emit more than once. _ can't be used as a parameter name in Java 9 or above.
There isn't a way you can send "empty" notifications. RxJava wiki suggests using Object in case you want to explicitly ignore the emitted value. For example:
compositeDisposable.add(
Observable.interval(500L, TimeUnit.MILLISECONDS)
.timeInterval()
.observeOn(AndroidSchedulers.mainThread())
.flatMap(longTimed -> {
if (emissionBoolean) {
//todo: find an observable that can emit 0 arguments
return Observable.just(new Object());
}
return Observable.empty()
})
.subscribe(object -> { // you still need to declare though.
onTick();
}));
Also, the code can be cleaner if you can switch to Kotlin, because in Kotlin you don't need to explicitly declare the name of the parameter if there is only one parameter.
compositeDisposable.add(
Observable.interval(500L, TimeUnit.MILLISECONDS)
.timeInterval()
.observeOn(AndroidSchedulers.mainThread())
.flatMap {
if (emissionBoolean) {
Observable.just(Any())
} else {
Observable.empty<Any>()
}}
.subscribe {
onTick()
}

RxJava 2 Observable that onComplete resubmits itself

I'm new with RxJava. I'm trying to create an observable that when it completes it will start all over again until I call dispose, but I'm facing an OutofMemory error after a while, below is a simplified example of what I'm trying to do
public void start() throws RuntimeException {
log.info("\t * Starting {} Managed Service...", getClass().getSimpleName());
try {
executeObserve();
log.info("\t * Starting {} Managed Service...OK!", getClass().getSimpleName());
} catch (Exception e) {
log.info("Managed Service {} FAILED! Reason is {} ", getClass().getSimpleName(), e.getMessage(), e);
}
}
start is invoked at the initialization phase once, the executeObserve is as follows (in a simplified form..). Notice that on the onComplete I "resubmit" executeObserve
public void executeObserve() throws RuntimeException {
Observable<Book> booksObserve
= manager.getAsObservable();
booksObserve
.map(Book::getAllOrders)
.flatMap(Observable::fromIterable)
.toList()
.subscribeOn(Schedulers.io())
.subscribe(collectedISBN ->
Observable.fromIterable(collectedISBN)
.buffer(10)
// ...some more steps here...
.toList()
.toObservable()
// resubmit
.doOnComplete(this::executeObserve)
.subscribe(validISBN -> {
// do something with the valid ones
})
)
);
}
My guess is that this is not the way to go if I want to resubmit my tasks but it was not possible to find any documentation.
the booksObserve is implemented as follows
public Observable<Book> getAsObservable() {
return Observable.create(e -> {
try (CloseableResultSet<Book> rs = (CloseableResultSet<Book>) datasource.retrieveAll())) {
for (Book r : rs) {
e.onNext(r);
}
e.onComplete();
} catch (Exception ex) {
e.onError(ex);
}
});
}
What is the correct way to constantly resubmit an operation until we call dispose or equivalent? I'm using RxJava 2

You have created an endless recursion, the loop will create more and more resources and sometime it will blow with OutOfMemory/Stack overflow exception.
In order to repeat the Observable work you should use repeat() operator, it will resubscribes to the Observable when it receives onComplete().
Besides that, some general comments on your code:
why are you nesting the second Observable inside the subscriber? you are breaking the chain, you can just continue the chain instead of creating new Observable at the Subscriber.
Moreover, it's seems (assuming Observable.fromIterable(collectedBets) using the collectedISBN that gets with the onNext() o.w. from where does it comes?) you're collecting all items to a list, and then flatting it again using from iterable, so it's seems you can just continue on the stream , something like that:
booksObserve
.map(Book::getAllOrders)
.flatMap(Observable::fromIterable)
.buffer(10)
// ...some more steps here...
.toList()
.toObservable()
// resubmit
.doOnComplete(this::executeObserve)
.subscribeOn(Schedulers.io())
.subscribe(validISBN -> {
// do something with the valid ones
});
Anyhow, with the nested Observable, the repeat() operator will just repeat the nested one, and not the entire stream (which is what you want) as it is not connected to it.

In continuation to my question the repeat as #yosriz suggested is the proper way to go, the following simple snippet demonstrates that the observable source will be called on each repeat
Observable<Integer> recursiveObservable = Observable.create(emitter -> {
System.out.println("Calling to emit data");
Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 0).forEach(emitter::onNext);
emitter.onComplete();
});
recursiveObservable
.buffer(2)
.repeat()
.subscribe(integers -> {
System.out.println(integers);
TimeUnit.SECONDS.sleep(1);
});

How to handle error thown in RxJava chain

I am trying to execute following code
testRepository
.exists(data)
.flatMap(x -> {
if (x==null) {
return Observable.error(new Exception("Error"));
}
return Observable.just(x);
})
.flatMap(x -> testRepository.create(x))
.flatMap(x -> {
return Observable.just(x);
});
This code works when no error is thrown in first map. But in case when error is thrown it just hangs.
What is wrong here?
Thanks

You haven't declared error properly, you shouldn't perform explicit converting to error observable via flatMap, because in case positive state you create new observable for each item and combine them after all.
You may just use
.doOnNext(x -> {
if (x == null) throw new IllegalStateException("null item error");
})
And this exception will interrupt stream and will be properly handled in onError callback of subscription.
Last instruction also doesn't make sense, because you convert each element to a single item observable and then combine them back to a similar stream.
Ps: also it needs to be called .subscribe somewhere, but i think it is meant.