Develop Reference

Wait for a webclient remote flux with awaitility

I am testing a webclient that returns a flux and I need to wait for it to initialise properly. Like this
I setup a flux as null
private Flux<Event> events = null;
Then call a webclient to get the Flux from a remote URL
Flux<String> events = getFlux(guid);
The webclient is
WebClient client; // already setup with headers and URL
public Flux<String> getFlux(String guid) {
return client.get()
.uri(Props.getBaseEndpoint() + "?id=" + guid)
.retrieve()
.onStatus(status -> status.value() == 401,clientResponse -> Mono.empty())
.bodyToFlux(String.class)
.timeout(Duration.ofSeconds(Props.getTimeout()));
}
The getFlux method appears to return before the Flux is completely initialised. So I want to wait a couple of seconds for it:
Awaitility.await().atMost(5, TimeUnit.SECONDS).until(isFluxInitialised());
where something like:
public Callable<Boolean> isFluxInitialised() {
return new Callable<Boolean>() {
public Boolean call() throws Exception {
if (events != null)
return true;
return false;
}
};
}
Waiting for the Flux to be not null still causes a race condition in the test. I can't figure out what to wait for so that the getFlux has returned an initialised Flux that can then be subscribed to. The test continues with a subscription to the flux as below but finishes before the test data that's sent to the remote endpoint can arrive in the subscription.
events.subscribe(e -> Logs.Info("event: " + e));
Here's the intellisense

Not sure I understand the logic of the isFluxInitialised but looking at the description you could be confused by Assembly vs Subscription time. Also, please note that subscribe is not synchronous operation and your program could exit before results are available.
I would suggest to start with unit test using StepVerifier to make sure your flow is correct.
StepVerifier.create(getFlux(...))
.expectNextCount(count)
.verifyComplete();
If you need to wait until Flux is complete in your logic you can use common pattern using CountDownLatch. The same can be achieved with Awaitility if you like.
CountDownLatch completionLatch = new CountDownLatch(1);
getFlux(...)
.doOnComplete(completionLatch::countDown)
.doOnNext(e -> Logs.Info("event: " + e))
.subscribe();
completionLatch.await();

There is no reason to introduce blocking operator blockFirst() into the flow. Still not sure about the use case but technically you are trying to wait for the first element from the Flux. The same could be achieved without blocking
AtomicBoolean elementAvailable = new AtomicBoolean();
getFlux()
.doOnNext(rec -> elementAvailable.set(true))
.subscribe();
Awaitility.await().atMost(5, TimeUnit.SECONDS).until(elementAvailable::get);

Thanks to #Alex answer about assembly v subscription time. That's the problem. I actually got the awaitility to work properly by moving it off waiting for "assembly" time (which I couldn't get to work) and instead to wait for the 1st subscription, by using blockFirst() as below:
Awaitility.await().atMost(5, TimeUnit.SECONDS).until(isFluxInitialised());
and
public Callable<Boolean> isFluxInitialised() {
return new Callable<Boolean>() {
public Boolean call() throws Exception {
if (events.blockFirst() != null)
return true;
return false;
}
};
}

What's the easiest way to wait for Mono completion in the background?

We are given a Mono, that's handling some action(say a database update), and returns a value.
We want to add that Mono(transformed) to a special list that contains actions to be completed for example during shutdown.
That mono may be eagerly subscribed after adding to the list, to start processing now, or .subscribe() might not be called meaning it will be only subscribed during shutdown.
During shutdown we can iterate on the list in the following way:
for (Mono mono : specialList) {
Object value = mono.block(); // (do something with value)
}
How to transform the original Mono such that when shutdown code executes, and Mono was previously subscribed(), the action will not be triggered again but instead it will either wait for it to complete or replay it's stored return value?

OK, looks like it is as simple as calling mono.cache(), so this is how I used it in practice
public Mono<Void> addShutdownMono(Mono<Void> mono) {
mono = mono.cache();
Mono<Void> newMono = mono.doFinally(signal -> shutdownMonos.remove(mono));
shutdownMonos.add(mono);
return newMono;
}
public Function<Mono<Void>,Mono<Void>> asShutdownAwaitable() {
return mono -> addShutdownMono(mono);
}
database.doSomeAction()
.as(asShutdownAwaitable)
.subscribe() // Or don't subscribe at all, deferring until shutdown
Here is the actual shutdown code.
It was also important to me that they execute in order of being added, if user chose not to eagerly subscribe them, that's reason for Flux.concat instead of Flux.merge.
public void shutdown() {
Flux.concat(Lists.transform(new ArrayList<>(shutdownMonos), mono -> mono.onErrorResume(err -> {
logger.error("Async exception during shutdown, ignoring", err);
return Mono.empty();
}))
).blockLast();
}

Limit for `onErrorContinue(...)` in Flux?

I have a (possibly infinite) Flux source that is supposed to first store each message (e.g. into a database) and then asynchronously forward the messages (e.g. using Spring WebClient).
The forward(s) in case of failure are supposed to log an error, without completing the source Flux.
I however realized that forward(s) wihtin the flow (flatMap(...)) block execution of the source Flux after exactly 256 messages that cause exceptions (e.g. reactor.retry.RetryExhaustedException).
Representative example that fails in the assert since only 256 messages are processed:
#Test
#SneakyThrows
public void sourceBlockAfter256Exceptions() {
int numberOfRequests = 500;
Set<Integer> sink = new HashSet<>();
Flux
.fromStream(IntStream.range(0, numberOfRequests).boxed())
.map(sink::add)
.flatMap(i -> Mono
// normally the forwards are contained here e.g. by means of Mono.when(...).thenReturn(...).retryWhen(...):
.error(new Exception("any"))
)
.onErrorContinue((throwable, o) -> log.error("Error", throwable))
.subscribe();
Thread.sleep(3000);
Assertions.assertEquals(numberOfRequests, sink.size());
}
Doing the forward within the subscribe(...) doesn't block the source Flux but that's certainly no solution, since I don't possibly want to lose messages.
Questions:
What has happened here? (probably related to some state stored in just one bit)
How can I do this correctly?
EDIT:
According to the discussion below I've constructed an example that uses FluxMessageChannel (which up to my understanding is made for infinite streams and definitly not expected to block after 256 Errors) and has exactly the same behaviour:
#Test
#SneakyThrows
public void maxConnectionWithChannelTest() {
int numberOfRequests = 500;
Set<Integer> sink = new HashSet<>();
FluxMessageChannel fluxMessageChannel = MessageChannels.flux().get();
fluxMessageChannel.subscribeTo(
Flux
.fromStream(IntStream
.range(0, numberOfRequests).boxed()
.map(i -> MessageBuilder.withPayload(i).build())
)
.map(Message::getPayload)
.map(sink::add)
.flatMap(i -> Mono.error(new Exception("whatever")))
);
Flux
.from(fluxMessageChannel)
.subscribe();
Thread.sleep(3000);
Assert.assertEquals(numberOfRequests, sink.size());
}
EDIT:
I just raised an issue in the reactor core project: https://github.com/reactor/reactor-core/issues/2011

Using Java `CompletablyFuture.exceptionally` to optionally continue through exceptions

I have a series of asynchronous tasks chained together using Java CompletableFutures. The code looks something like this:
CompletableFuture<Result> doTasks(final TaskId id) {
return firstTask.workAsync(id)
.thenComposeAsync(__ -> secondTask.workAsync(id))
.thenComposeAsync(__ -> thirdTask.workAsync(id))
.thenApplyAsync(__ -> fourthTask.workAsync(id));
}
However, firstTask.workAsync throws an exception indicating that the work has already been completed, which is OK in this situation, so I would like to just ignore it and continue through the chain.
Of course, I could just wrap that bit in a separate function where I can handle the exception, but is there a way to handle it directly in the CompletableFuture chain and continue to throw all other exceptions?
A co-worker suggested I use CompletableFuture.exceptionally, but all of the examples online that I see are totally useless and just return null, which looks like it would kill the chain. How would I use that in this case?

CompletableFuture.exceptionally can be used to continue when getting an exception in a CompletableFuture. In a nutshell, you need to check the type of the exception, and if it's an exception you want to continue on, you can return a new CompletableFuture, which can be empty since the result is not used down the chain.
CompletableFuture<Result> doTasks(final TaskId id) {
return firstTask.workAsync(id)
.exceptionally(t -> {
// Will continue down the chain if matches
if (t instanceof TotallyOkException) {
return null;
}
// This will throw an ExecutionException. I convert it to a RuntimeException here
// because I don't want to add throws statements up the chain.
throw new RuntimeException(t);
})
.thenComposeAsync(__ -> secondTask.workAsync(id))
.thenComposeAsync(__ -> thirdTask.workAsync(id))
.thenApplyAsync(__ -> fourthTask.workAsync(id));
}
In this case, it will throw all non-TotallyOkException exceptions.

Returning null in your exceptionally function will not, in itself, kill the chain. The only way it will kill the chain is a result of lack of null handling in the downstream function and causing a NullPointerException.
Your exceptionally function can be set up to handle some types of exception and not others. For example:
return firstTask.workAsync(id)
.thenComposeAsync(firstResult -> secondTask.workAsync(id))
.exceptionally(t -> {
if (t instanceof TransientException) {
return getUsingBackupMethod(id);
}
throw new RuntimeException(t);
});
This exceptionally function will (effectively) catch an exception thrown from either of the first two tasks.

Spring Integration Java DSL: file inbound exception propagation

Given the following file inbound:
IntegrationFlows.from(s -> s
.file(directory, new LastModifiedFileComparator())
.patternFilter(inputFileNamePattern)
.preventDuplicates(),
e -> e.poller(p -> p.trigger(filePollerTrigger))
)
and the trigger that throws exception in case certain time was overreached, how does one receives exception that was thrown?
Will it appear in flow exception channel or in inbound specific error handler?
What is the correct way to deal with it in java dsl?
Thanks in advance.

The exception thrown from the trigger.nextExecutionTime() causes the polling task to be stopped, or not rescheduled if that sounds better:
public ScheduledFuture<?> schedule() {
synchronized (this.triggerContextMonitor) {
this.scheduledExecutionTime = this.trigger.nextExecutionTime(this.triggerContext);
if (this.scheduledExecutionTime == null) {
return null;
}
long initialDelay = this.scheduledExecutionTime.getTime() - System.currentTimeMillis();
this.currentFuture = this.executor.schedule(this, initialDelay, TimeUnit.MILLISECONDS);
return this;
}
}
As you see by code it is fully equivalent to the null from the trigger. We just exit from the rescheduling loop.
Consider to implement the exception handling logic in the custom Trigger as a wrapper around that target one. For example ErrorHandlingTrigger.