Develop Reference

Wait for a webclient remote flux with awaitility

I am testing a webclient that returns a flux and I need to wait for it to initialise properly. Like this
I setup a flux as null
private Flux<Event> events = null;
Then call a webclient to get the Flux from a remote URL
Flux<String> events = getFlux(guid);
The webclient is
WebClient client; // already setup with headers and URL
public Flux<String> getFlux(String guid) {
return client.get()
.uri(Props.getBaseEndpoint() + "?id=" + guid)
.retrieve()
.onStatus(status -> status.value() == 401,clientResponse -> Mono.empty())
.bodyToFlux(String.class)
.timeout(Duration.ofSeconds(Props.getTimeout()));
}
The getFlux method appears to return before the Flux is completely initialised. So I want to wait a couple of seconds for it:
Awaitility.await().atMost(5, TimeUnit.SECONDS).until(isFluxInitialised());
where something like:
public Callable<Boolean> isFluxInitialised() {
return new Callable<Boolean>() {
public Boolean call() throws Exception {
if (events != null)
return true;
return false;
}
};
}
Waiting for the Flux to be not null still causes a race condition in the test. I can't figure out what to wait for so that the getFlux has returned an initialised Flux that can then be subscribed to. The test continues with a subscription to the flux as below but finishes before the test data that's sent to the remote endpoint can arrive in the subscription.
events.subscribe(e -> Logs.Info("event: " + e));
Here's the intellisense

Not sure I understand the logic of the isFluxInitialised but looking at the description you could be confused by Assembly vs Subscription time. Also, please note that subscribe is not synchronous operation and your program could exit before results are available.
I would suggest to start with unit test using StepVerifier to make sure your flow is correct.
StepVerifier.create(getFlux(...))
.expectNextCount(count)
.verifyComplete();
If you need to wait until Flux is complete in your logic you can use common pattern using CountDownLatch. The same can be achieved with Awaitility if you like.
CountDownLatch completionLatch = new CountDownLatch(1);
getFlux(...)
.doOnComplete(completionLatch::countDown)
.doOnNext(e -> Logs.Info("event: " + e))
.subscribe();
completionLatch.await();

There is no reason to introduce blocking operator blockFirst() into the flow. Still not sure about the use case but technically you are trying to wait for the first element from the Flux. The same could be achieved without blocking
AtomicBoolean elementAvailable = new AtomicBoolean();
getFlux()
.doOnNext(rec -> elementAvailable.set(true))
.subscribe();
Awaitility.await().atMost(5, TimeUnit.SECONDS).until(elementAvailable::get);

Thanks to #Alex answer about assembly v subscription time. That's the problem. I actually got the awaitility to work properly by moving it off waiting for "assembly" time (which I couldn't get to work) and instead to wait for the 1st subscription, by using blockFirst() as below:
Awaitility.await().atMost(5, TimeUnit.SECONDS).until(isFluxInitialised());
and
public Callable<Boolean> isFluxInitialised() {
return new Callable<Boolean>() {
public Boolean call() throws Exception {
if (events.blockFirst() != null)
return true;
return false;
}
};
}

What's the easiest way to wait for Mono completion in the background?

We are given a Mono, that's handling some action(say a database update), and returns a value.
We want to add that Mono(transformed) to a special list that contains actions to be completed for example during shutdown.
That mono may be eagerly subscribed after adding to the list, to start processing now, or .subscribe() might not be called meaning it will be only subscribed during shutdown.
During shutdown we can iterate on the list in the following way:
for (Mono mono : specialList) {
Object value = mono.block(); // (do something with value)
}
How to transform the original Mono such that when shutdown code executes, and Mono was previously subscribed(), the action will not be triggered again but instead it will either wait for it to complete or replay it's stored return value?

OK, looks like it is as simple as calling mono.cache(), so this is how I used it in practice
public Mono<Void> addShutdownMono(Mono<Void> mono) {
mono = mono.cache();
Mono<Void> newMono = mono.doFinally(signal -> shutdownMonos.remove(mono));
shutdownMonos.add(mono);
return newMono;
}
public Function<Mono<Void>,Mono<Void>> asShutdownAwaitable() {
return mono -> addShutdownMono(mono);
}
database.doSomeAction()
.as(asShutdownAwaitable)
.subscribe() // Or don't subscribe at all, deferring until shutdown
Here is the actual shutdown code.
It was also important to me that they execute in order of being added, if user chose not to eagerly subscribe them, that's reason for Flux.concat instead of Flux.merge.
public void shutdown() {
Flux.concat(Lists.transform(new ArrayList<>(shutdownMonos), mono -> mono.onErrorResume(err -> {
logger.error("Async exception during shutdown, ignoring", err);
return Mono.empty();
}))
).blockLast();
}

Mono switchIfEmpty is always called

I have a bit more complex use case in Spring Gateway which is based on WebFlux and I ran into small issue with Mono usage. Long story short, my switchIfEmpty is called even if not required. I prepared small example which allows me to reproduce this problem:
public class ReactiveTest {
#Test
void test1() {
isOdd(2)
.flatMap(this::onNotEmpty)
.switchIfEmpty(Mono.defer(this::onEmpty))
.block();
}
Mono<String> isOdd(Integer number) {
return number % 2 != 0 ? Mono.just("Yes") : Mono.empty();
}
Mono<Void> onNotEmpty(String value) {
System.out.println("Value uppercased " + value.toUpperCase());
return Mono.empty();
}
Mono<Void> onEmpty() {
System.out.println("Value not present, this shouldn't been called if value was odd");
return Mono.empty();
}
}
I hope this is pretty self-explanatory, but just to be safe:
isOdd(Integer number) may produce Mono with data or empty Mono
I expect onNotEmpty to be called only if previous Mono had data
I expect onEmpty to be called only if isOdd produced empty Mono
Unfortunatelly, both onNotEmpty and onEmpty are called all the time, regardless if I pass odd or even number to isOdd.
How can I make sure that onEmpty is called only when isOdd produced Mono.empty()?

The onNotEmpty(String value) method is always returning Mono.empty(), meaning that .switchIfEmpty(Mono.defer(this::onEmpty)) will always be call either because isOdd(2) is already an empty Mono or because onNotEmpty(String value) method was called and returned an empty Mono.
In order to avoid this, you need to change your onNotEmpty(String value) method to return something else than an empty Mono.
Additionally, please avoid using block() since this defeats the whole purpose of using Spring WebFlux by blocking the thread waiting for something to be emitted by the reactive chain.

Call a WebService and a REST API using JDK8 Streams and CompletableFuture

I have a SOAP call that I need to make and then process the results from the SOAP call in a REST call. Each set of calls is based on a batch of records. I am getting completely lost in trying to get this to run using JDK8 streams as asynchronous as possible. How can I accomplish this?
SOAP Call:
CompletableFuture<Stream<Product>> getProducts(final Set<String> criteria)
{
return supplyAsync(() -> {
...
return service.findProducts(request);
}, EXECUTOR_THREAD_POOL);
}
REST Call:
final CompletableFuture<Stream<Result>> validateProducts(final Stream<Product> products)
{
return supplyAsync(() -> service
.submitProducts(products, false)
.stream(), EXECUTOR_THREAD_POOL);
}
I am trying to invoke the SOAP call, pass the result into the REST call, and collect the results using a JDK8 stream. Each SOAP->REST call is a "set" of records (or batch) similar to paging. (this is totally not working right now but just an example).
#Test
public void should_execute_validations()
{
final Set<String> samples = generateSamples();
//Prepare paging...
final int total = samples.size();
final int pages = getPages(total);
log.debug("Items: {} / Pages: {}", total, pages);
final Stopwatch stopwatch = createStarted();
final Set<Result> results = range(0, pages)
.mapToObj(index -> {
final Set<String> subset = subset(index, samples);
return getProducts(subset)
.thenApply(this::validateProducts);
})
.flatMap(CompletableFuture::join)
.collect(toSet());
log.debug("Executed {} calls in {}", pages, stopwatch.stop());
assertThat(results, notNullValue());
}

I think there are two usage that are incorrect in your example: thenApply and join.
To chain the 1st call (SOAP) and the 2nd call (REST), you need to use thenCompose instead of thenApply. This is because method "validateProducts" returns completable futures, using "thenApply" will create CompletableFuture<CompletableFuture<Stream<Result>>> in your stream mapping. But what you need is probably CompletableFuture<Stream<Result>>. Using thenCompose can resolve this problem, because it is analogous to "Optional.flatMap" or "Stream.flatMap":
.mapToObj(index -> {
final Set<String> subset = subset(index, samples);
return getProducts(subset)
.thenCompose(this::validateProducts);
})
The 2nd incorrect usage is join. Using join blocks the current thread waiting for the result of that CompletableFuture. In your cases, there are N completable futures, where N is the number of pages. Instead of waiting them one by one, the better solution is to wait all the them use CompletableFuture.allOf(...). This method returns a new CompletableFuture that is completed when all of the given CompletableFutures complete. So I suggest that you modify your stream usage and return a list of futures. Then, wait the completion. And finally, retrieve the results:
List<CompletableFuture<Stream<Result>>> futures = range(0, pages)
.mapToObj(index -> {
final Set<String> subset = subset(index, samples);
return getProducts(subset).thenCompose(this::validateProducts);
})
.collect(Collectors.toList());
CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join();
for (CompletableFuture<Stream<Result>> cf : futures) {
// TODO Handle the results and exceptions here
}
You can see the complete program on GitHub.

Use Reactor Core Java to send to browser the status of a long executing task, periodically

I have been working with some Reactor Core Java, because I want to figure out if this is possible to solve one problem I currently have using this framework.
At present I have a long, executing job that takes about 40-50 minutes to complete. The method looks more or less like this:
public void doLongTask(List<Something> list){
//instructions.
for(Something sm : list){
if(condition){
executeLongOperation();
}
//instructions
if(condition){
executeLongOperation();
}
}
}
in my controller I have something like this:
#GetMapping(path = "/integersReactor", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
#ResponseBody
public Flux<Integer> getIntegersReactor(){
logger.debug("Request getIntegersReactor initialized.");
return simpleSearchService.getIntegersReactor();
}
and in the service layer I have something like this:
#Override
public Flux<Integer> getIntegersReactor(){
return Flux.range(0, Integer.MAX_VALUE);
}
this is just a placeholder that I am using as a proof of concept. My real intentions are to somehow return a Flux of some object that I will define myself, this object will have a few fields that I will use to tell the consumer the status of the job.
Now, things get somewhat complicated now because I would like to send updates as the executeLongOperation(); are executed, and somehow instead of returning a flux of Integers, return a flux of an object that uses the return of executeLongOperation();
Can this be acomplished with Flux? How can I leverage Reactor Core java to push the return values of all of the times executeLongOperation(); is executed into a reactive stream that can be passed to the controller the same way that getIntegersReactor() does it in my example?

Yes it should be possible, but since the executeLongOperation is blocking, it will need to be offset on a dedicated thread (which reduces the benefits you get from a top-to-bottom reactive implementation).
Change your doLongTask to return a Flux<Foo>, make it concatenate Monos that wrap executeLongOperation on a dedicated thread (or better yet, change the executeLongOperation itself to return a Mono<Foo> and do the wrapping internally and subscribeOn another thread internally). Something like:
public Flux<Foo> doLongTask(List<Something> list) {
return Flux.fromIterable(list)
//ensure `Something` are published on a dedicated thread on which
//we can block
.publishOn(Schedulers.elastic()) //maybe a dedicated Scheduler?
//for each `Something`, perform the work
.flatMap(sm -> {
//in case condition is false, we'll avoid long running task
Flux<Foo> work = Flux.empty();
//start declaring the work depending on conditions
if(condition) {
Mono<Foo> op = Mono.fromCallable(this::executeLongOperation);
work = conditional.concatWith(op);
}
//all other instructions should preferably be non-blocking
//but since we're on a dedicated thread at this point, it should be ok
if(condition) {
Mono<Foo> op = Mono.fromCallable(this::executeLongOperation);
work = conditional.concatWith(op);
}
//let the flatMap trigger the work
return work;
});
}