Develop Reference

How to wait until List<Mono<List<Object>>> finishes?

It's my first time working with webClient and I am wondering how to wait until List<Mono<>> finishes. I have the following code:
List<Address> addresses = collectAllAddresses(someObject);
List<Mono<List<AnotherAddress>>> monoResponses =
addresses.stream()
.map(address -> webClientGateway.findAddresses(userData, address.getFullAddress()))
.collect(Collectors.toList());
Mono.when(monoResponses).block();
log.info("mono responses");
monoResponses.stream()
.flatMap(it -> Objects.requireNonNull(it.block()).stream()).forEach(it -> log.info("mono responses + {}", it));
and the following findAddresses method:
public Mono<List<AnotherAddress>> findAddresses(UserData userData, String fullAddress) {
if (StringUtils.isEmpty(fullAddress)) {
log.info("Address is empty that why we return Mono.just(Collections.emptyList()");
return Mono.just(Collections.emptyList());
}
return webClient.get()
.uri(path, uri -> uri.queryParam("query", fullAddress).queryParam("count", 1).build())
.header("someHeader", someHeader)
.retrieve()
.bodyToMono(new ParameterizedTypeReference<List<AnotherAddress>>() {
})
.doOnError(e -> log.error("Error occurred!", e));
}
but every time I execute it I always get list of empty objects, I mean I get List but every object in that list is empty (every field of class AnotherAddress is null). What can be wrong?
UDP: more explanations:
I have two microservices. In one microservice (that return another address) there is RestController that sends anotherAddress. In another microservice I want to use WebClient (instead of using threadPool with many threads) to call the RestController from previous microservice. When I have previous implementation for function webClientGateway.findAddresses(userData, address.getFullAddress()) and it returns Mono<List> I tested it and immediately after calling function I call block on result and it works. But now I have following situation, I have many addresses (maybe 5 or 10) and I want send async request for every address and wait until latest finishes and after that I want to do another operation, but instead of getting fullfielded AnotherAddress instance, I am getting 5 empty AnotherAddress instances (every field is null)

Use a Flux instead of a Mono, e.g. something like (untested):
public Flux<AnotherAddress> findAddresses(UserData userData, String fullAddress) {
if (StringUtils.isEmpty(fullAddress)) {
log.info("Address is empty that why we return Mono.just(Collections.emptyList()");
return Flux.empty();
}
return webClient.get()
.uri(path, uri -> uri.queryParam("query", fullAddress).queryParam("count", 1).build())
.header("someHeader", someHeader)
.retrieve()
.bodyToFlux(AnotherAddress.class)
.doOnError(e -> log.error("Error occurred!", e));
}
If you don't need the AnotherAddress list grouped by address the you could use something like (untested):
Flux<AnotherAddress> anotherAddressFlux= Flux.fromIterable(addresses)
.flatMap(address -> webClientGateway.findAddresses(userData, address.getFullAddress()));
If you want to block you can use:
List<AnotherAddress> anotherAddressList = anotherAddressFlux.collectList().block();

Convert traditional loop of invoking weclient into non blocking way

I am new to reactive programming and I want to transform the following code into non blocking way.
For the sake of simplicity, I created a sample pseudo code based from my original code. Any help will be appreciated.
public Mono<Response> getResponse(List<Provider> providers) {
for (Provider provider : providers) {
Response response = provider.invokeHttpCall().block();
if(response.getMessage() == "Success") {
return Mono.just(response);
}
continue;
}
return Mono.empty();
}
provider.invokeHttpCall() method
#Override
public Mono<Response> invokeHttpCall(){
WebClient webClient = WebClient.create();
return webClient.post()
.uri("/provider").accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToMono(Response.class);
}
I tried several tactics to implement this, but still no luck. Either all providers are invoked or I need to block the webclient thread.

Flux.fromIterable(providers)
.concatMap(Provider::invokeHttpCall) // ensures providers are called sequentially
.filter(response -> response.getMessage().equals("Success"))
.next()

reactive is a kind of Stream. Please think it as a Stream and program it reactively.
I give you such followed code.
Firstly, use Flux.fromIterable() to create a flux stream from a List.
Next, use flatmap() and Lambda fuction to emit the invoke into another new thread.
use method filterWhen() and Lambda to get the "Success" response and just get the first "Success" elements. See filterwhen api Doc.
Finally, just use Mono.from() to wrap the Flux and then return the Mono type.
public Mono<Response> getResponse(List<Provider> providers) {
return Mono.from(Flux.fromIterable(providers)
.flatmap(provider ->
Mono.defer(() -> provider.invokeHttpCall())
.filterWhen(response -> response.getMessage() == "Success");
}
if you want to see result and println().
Just use .subsribe() method to excute it.
getResponse.subsribe(System.out::println);

spring webclient: retry with backoff on specific error

i'd like to retry the request 3 times after waiting 10sec when response is 5xx. but i don't see a method that I can use. On object
WebClient.builder()
.baseUrl("...").build().post()
.retrieve().bodyToMono(...)
i can see methods:
retrying on condition with retry count but no delay
.retry(3, {it is WebClientResponseException && it.statusCode.is5xxServerError} )
retrying with backoff and number of times but no condition
.retryBackoff
there is also a retryWhen but i'm not sure how to use it

With reactor-extra you could do it like:
.retryWhen(Retry.onlyIf(this::is5xxServerError)
.fixedBackoff(Duration.ofSeconds(10))
.retryMax(3))
private boolean is5xxServerError(RetryContext<Object> retryContext) {
return retryContext.exception() instanceof WebClientResponseException &&
((WebClientResponseException) retryContext.exception()).getStatusCode().is5xxServerError();
}
Update:
With new API the same solution will be:
.retryWhen(Retry.fixedDelay(3, Duration.ofSeconds(10))
.filter(this::is5xxServerError));
//...
private boolean is5xxServerError(Throwable throwable) {
return throwable instanceof WebClientResponseException &&
((WebClientResponseException) throwable).getStatusCode().is5xxServerError();
}

You can do this taking the following approach:
Use the exchange() method to obtain the response without an exception, and then throw a specific (custom) exception on a 5xx response (this differs from retrieve() which will always throw WebClientResponseException with either a 4xx or 5xx status);
Intercept this specific exception in your retry logic;
Use reactor-extra - it contains a nice way to use retryWhen() for more complex & specific retries. You can then specify a random backoff retry that starts after 10 seconds, goes up to an arbitrary time and tries a maximum of 3 times. (Or you can use the other available methods to pick a different strategy of course.)
For example:
//...webclient
.exchange()
.flatMap(clientResponse -> {
if (clientResponse.statusCode().is5xxServerError()) {
return Mono.error(new ServerErrorException());
} else {
//Any further processing
}
}).retryWhen(
Retry.anyOf(ServerErrorException.class)
.randomBackoff(Duration.ofSeconds(10), Duration.ofHours(1))
.maxRetries(3)
)
);

the retryWhen with Retry.anyOf and Retry.onlyIf are deprecated I assume. I found this approach useful, and it allows us to process and throw a User defined exception.
for example :
retryWhen(Retry.backoff(3, Duration.of(2, ChronoUnit.SECONDS))
.filter(error -> error instanceof UserDefinedException/AnyOtherException)
.onRetryExhaustedThrow((retryBackoffSpec, retrySignal) ->
new UserDefinedException(retrySignal.failure().getMessage())))

// ...
.retryWhen(
backoff(maxAttempts, minBackoff)
.filter(throwable -> ((WebClientResponseException) throwable).getStatusCode().is5xxServerError()))
// ...

Adding only withThrowable to your existing code can make it work. This has worked for me. You can try something like this :
For example :
.retryWhen(withThrowable(Retry.any()
.doOnRetry(e -> log
.debug("Retrying to data for {} due to exception: {}", employeeId, e.exception().getMessage()))
.retryMax(config.getServices().getRetryAttempts())
.backoff(Backoff.fixed(Duration.ofSeconds(config.getServices().getRetryBackoffSeconds())))))

here's how i do it:
.retryWhen(retryBackoffSpec())
private RetryBackoffSpec retryBackoffSpec() {
return Retry.backoff(RETRY_ATTEMPTS, Duration.ofSeconds(RETRY_DELAY))
.filter(throwable -> throwable instanceof yourException);
}

how to solve the problem with put one data in mono to another mono in Spring Webflux?

I want to get a string data from another server by webclient object, and put it to another Mono object. But in a webclient, only readable that in .subscribe().
Because responseBody.subscribe() method is async, method test() will be return result object with empty message field before responseBody.subscribe() executed.
Of course, I knew that if I return responseBody object instead of result object, there is no problem. But I want to return not a responseBody object but result object with not empty field of message.
I want to return result when responseBody's subscribe() is completed.
How to change my code?
Please help me.
public Mono<ResultVO> test() {
Mono<ResultVO> result = Mono.just(new ResultVO());
WebClient client = webClientBuilder.baseUrl("http://XXXXXX").build();
Mono<String> responseBody = client.get().uri("/aaaa/bbbbb").retrieve().bodyToMono(String.class);
responseBody.subscribe( s -> {
result.subscribe(g -> g.setMessage(s));
});
return result;
}
...
#Data
public class ResultVO {
private long timestamp;
private String ip;
private String message;
...
}
I expect like this
{
"timestamp": 1566662695203,
"ip": "192.168.1.1",
"message": "c0db76f6-4eb5-4f84-be8d-018d53b453bb"
}
But result data is,
{
"timestamp": 1566662695203,
"ip": "192.168.1.1",
"message": ""
}

Putting this kind of logic into the subscribe method is not recommended, it can easily lead to 'callback hell' and eventually unmaintainable code. Also, I don't see the caller of the shared test method, but chances are that one of the Monos is subscribed twice, which also leads to quite confusing behaviour.
Instead, to combine Monos you can use zip, zipWith, flatMap and a couple of other operators.
One solution with zipWith method:
public Mono<ResultVO> test()
{
WebClient client = WebClient.builder().baseUrl("http://XXXXXX").build();
// dummy representation of another data source (db query, web service call...)
Mono<ResultVO> result = Mono.just(new ResultVO());
Mono<String> responseBody = client.get().uri("/aaaa/bbbbb").retrieve().bodyToMono(String.class);
return result.zipWith(responseBody,
(resultObj, body) -> new ResultVO(resultObj.getTimestamp(), resultObj.getIp(), body));
}
Couple of other notes:
If you are returning JSON through a REST endpoint of your reactive WebFlux application, then you never need to subscribe manually, Spring will do that for you
Avoid using mutable objects (the ones which you modify after creation with setters), instead create new object, this will make your code easier to reason about and less prone to concurrency issues
Useful read about available Reactor operators

First of all, you hardly ever subscribe in your own application.
Think of it this way. Your server is a publisher, that means that your server fetches data and then publishes it to whomever wants it.
The subscriber is usually the end client, that could be a react application, an angular application or any client.
I think you need to read up on the basics of how to use webflux and reactive programming.
This is how to do what you are asking for, with as minimal changes to your code, we map what we fetched to what we want returned.
public Mono<ResultVO> test() {
final WebClient client = webClientBuilder
.baseUrl("http://XXXXXX").build();
return client.get()
.uri("/aaaa/bbbbb")
.retrieve()
.bodyToMono(String.class)
.map(message -> {
final ResultVO resultVO = new ResultVO();
resultVO.setMessage(message);
return resultVO;
}
);
}

Error handing in RxJava at the boundary between reactive and imperative code

Given an interface:
public interface FastlyRxApi {
#GET("/service/{service_id}/version/{version}/backend")
Observable<List<Backend>> listBackends(#Path("service_id") String serviceId, #Path("version") String versionId);
#PUT("/service/{service_id}/version/{version}/backend/{old_name}")
Observable<Backend> updateBackend(#Path("service_id") String serviceId, #Path("version") String version, #Path("old_name") String oldName, #Body Backend updatedBacked);
}
and some client code:
Integer expectedFirstByteTimeout = 10000;
// Use a final array to capture any problem found within our composed Observables
final FastlyEnvException[] t = new FastlyEnvException[1];
fastlyRxApi.listBackends(serviceId, newVersion)
.flatMap(Observable::fromIterable)
.filter(backend -> !expectedFirstByteTimeout.equals(backend.getFirstByteTimeout()))
.flatMap(backend -> {
backend.setFirstByteTimeout(expectedFirstByteTimeout);
return fastlyRxApi.updateBackend(serviceId, newVersion, backend.getName(), backend);
}).subscribe(ignore -> {
}, e -> {
t[0] = new FastlyEnvException("failed to configure backends", e);
});
if (t[0] != null) {
throw t[0];
}
Using a final array of FastlyEnvException to capture context for error handling feels like I'm doing something wrong, and missing some aspect.
Am I using a hammer rather than a screwdriver here; ie should I be using RxJava for this? It seems to give me a nice readable flow, apart from the error handling. What is the preferred idiom for doing this?

Use onErrorResumeNext:
.onErrorResumeNext(err ->
Observable.error(new FastlyEnvException("failed to configure backends", e)))
.toBlocking();
.subscribe();
Of note here is the .toBlocking(), this will make the Observable chain wait until it's complete.
Given that the subscribe() doesn't have an error handler, it will re-throw the exception.

Just omit error handler in your subscribe() and the error will be rethrown automatically.