Develop Reference

Java Reactor API: how to wait for an object to be modified by asynchronous calls to be completely modified before sending it back to the caller?

I'm totally new to the Java Reactor API.
I use a WebClient to retrieve data from an external webservice, which I then map to a DTO of class "LearnDetailDTO".
But before sending back this DTO, I have to modify it with data I get from another webservice. For this, I chain the calls with flatMap(). I get my data from the second webservice, but my DTO is returned before it is modified with the new data.
My problem is: how to wait until all calls to the second webservice are finished and the DTO is modified before sending it back to the caller?
Here is my code:
class Controller {
#GetMapping(value = "/learn/detail/", produces = MediaType.APPLICATION_JSON_VALUE)
public Mono<LearnDetailDTO> getLearnDetail() {
return getLearnDetailDTO();
}
private Mono<LearnDetailDTO> getLearnDetailDTO() {
WebClient client = WebClient.create("https://my_rest_webservice.com");
return client
.get()
.retrieve()
.bodyToMono(LearnDetailDTO.class)
.flatMap(learnDetailDTO -> {
LearnDetailDTO newDto = new LearnDetailDTO(learnDetailDTO );
for (GroupDTO group : newDto.getGroups()) {
String keyCode = group.getKeyCode();
for (GroupDetailDto detail : group.getGroupsDetailList()) {
adeService.getResourcesList(keyCode) // one asynchonous rest call to get resources
.flatMap(resource -> {
Long id = resource.getData().get(0).getId();
return adeService.getEventList(id); // another asynchronous rest call to get an events list with the resource coming from the previous call
})
.subscribe(event -> {
detail.setCreneaux(event.getData());
});
}
}
return Mono.just(newDto);
});
}
I tried to block() my call to adeservice.getEventList() instead of subscribe(), but I get the following error:
block()/blockFirst()/blockLast() are blocking, which is not supported
in thread reactor-http-nio-2
How to be sure that my newDTO object is complete before returning it ?

You should not mutate objects in subscribe. The function passed to subscribe will be called asynchronously in an unknown time in the future.
Subscribe should be considered a terminal operation, and should only serve to connect to other part of your system. It should not modify values inside the scope of your datastream.
What you want, is a pipeline that collects all events, and then map them to a dto with collected events.
As a rule of thumb your pipeline result must be composed of accumulated results in the operation chain. You should never have a "subscribe" in the middle of the operation chain, and you should never mutate an object with it.
I will provide a simplified example so you can take time to analyze the logic that can reach the goal: accumulate new values asynchronously in a single result. In this example, I've removed any notion of "detail" to connect directly groups to events, to simplify the overall code.
The snippet:
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
public class AccumulateProperly {
// Data object definitions
record Event(String data) {}
record Resource(int id) {}
record Group(String keyCode, List<Event> events) {
// When adding events, do not mute object directly. Instead, create a derived version
Group merge(List<Event> newEvents) {
var allEvents = new ArrayList<>(events);
allEvents.addAll(newEvents);
return new Group(keyCode, allEvents);
}
}
record MyDto(List<Group> groups) { }
static Flux<Resource> findResourcesByKeyCode(String keyCode) {
return Flux.just(new Resource(1), new Resource(2));
}
static Flux<Event> findEventById(int id) {
return Flux.just(
new Event("resource_"+id+"_event_1"),
new Event("resource_"+id+"_event_2")
);
}
public static void main(String[] args) {
MyDto dtoInstance = new MyDto(List.of(new Group("myGroup", List.of())));
System.out.println("INITIAL STATE:");
System.out.println(dtoInstance);
// Asynchronous operation pipeline
Mono<MyDto> dtoCompletionPipeline = Mono.just(dtoInstance)
.flatMap(dto -> Flux.fromIterable(dto.groups)
// for each group, find associated resources
.flatMap(group -> findResourcesByKeyCode(group.keyCode())
// For each resource, fetch its associated event
.flatMap(resource -> findEventById(resource.id()))
// Collect all events for the group
.collectList()
// accumulate collected events in a new instance of the group
.map(group::merge)
)
// Collect all groups after they've collected events
.collectList()
// Build a new dto instance from the completed set of groups
.map(completedGroups -> new MyDto(completedGroups))
);
// NOTE: block is here only because we are in a main function and that I want to print
// pipeline output before program extinction.
// Try to avoid block. Return your mono, or connect it to another Mono or Flux object using
// an operation like flatMap.
dtoInstance = dtoCompletionPipeline.block(Duration.ofSeconds(1));
System.out.println("OUTPUT STATE:");
System.out.println(dtoInstance);
}
}
Its output:
INITIAL STATE:
MyDto[groups=[Group[keyCode=myGroup, events=[]]]]
OUTPUT STATE:
MyDto[groups=[Group[keyCode=myGroup, events=[Event[data=resource_1_event_1], Event[data=resource_1_event_2], Event[data=resource_2_event_1], Event[data=resource_2_event_2]]]]]

Debounce similar requests with reactor-grpc

In order to offload my database, I would like to debounce similar requests in a gRPC service (say for instance that they share the same id part of the request) that serves an API which does not have strong requirements in terms of latency. I know how to do that with vanilla gRPC but I am sure what kind of API of Mono I can use.
The API calling directly the db looks like this:
public Mono<Blob> getBlob(
Mono<MyRequest> request) {
return request.
map(reader.getBlob(request.getId()));
I have a feeling I should use delaySubscription but then it does not seem that groupBy is part of the Mono API that gRPC services handle.

It's perfeclty ok to detect duplicates not using reactive operators:
// Guava cache as example.
private final Cache<String, Boolean> duplicatesCache = CacheBuilder.newBuilder()
.expireAfterWrite(Duration.ofMinutes(1))
.build();
public Mono<Blob> getBlob(Mono<MyRequest> request) {
return request.map(req -> {
var id = req.getId();
var cacheKey = extractSharedIdPart(id);
if (duplicatesCache.getIfPresent(cacheKey) == null) {
duplicatesCache.put(cacheKey, true);
return reader.getBlob(id);
} else {
return POISON_PILL; // Any object that represents debounce hit.
// Or use flatMap() + Mono.error() instead.
}
});
}
If for some reason you absolutely want to use reactive operators, then first you need to convert incoming grpc requests into Flux. This can be achieved using thirdparty libs like salesforce/reactive-grpc or directly:
class MyService extends MyServiceGrpc.MyServiceImplBase {
private FluxSink<Tuple2<MyRequest, StreamObserver<MyResponse>>> sink;
private Flux<Tuple2<MyRequest, StreamObserver<MyResponse>>> flux;
MyService() {
flux = Flux.create(sink -> this.sink = sink);
}
#Override
public void handleRequest(MyRequest request, StreamObserver<MyResponse> responseObserver) {
sink.next(Tuples.of(request, responseObserver));
}
Flux<Tuple2<MyRequest, StreamObserver<MyResponse>>> getFlux() {
return flux;
}
}
Next you subscribe to this flux and use operators you like:
public static void main(String[] args) {
var mySvc = new MyService();
var server = ServerBuilder.forPort(DEFAULT_PORT)
.addService(mySvc)
.build();
server.start();
mySvc.getFlux()
.groupBy(...your grouping logic...)
.flatMap(group -> {
return group.sampleTimeout(...your debounce logic...);
})
.flatMap(...your handling logic...)
.subscribe();
}
But beware of using groupBy with lots of distinct shared id parts:
The groups need to be drained and consumed downstream for groupBy to work correctly. Notably when the criteria produces a large amount of groups, it can lead to hanging if the groups are not suitably consumed downstream (eg. due to a flatMap with a maxConcurrency parameter that is set too low).

Spring WebFlux - how to determine when my client has finished working

I need to call certain API with multiple query params simultaneously, in order to do that I wanted to use reactive approach. I ended up with reactive client that is able to call endpoint based on passed SearchQuery, handle pagination of that response and call for remaining pages and returns Flux<Item>. So far it works fine, however what I need to do now is to:
Collect data for all search queries and save them as initial state
Once the initial data is collected, I need to start repeating those calls in small time intervals and validate each item against initial data. Basically, I need to find new items from here.
But I'm running out of options how to solve that, I came up with probably the dirties solution ever, but I bet there are much better ways to do that.
So first of all, this is relevant code of my client
public Flux<Item> collectData(final SearchQuery query) {
final var iteration = new int[]{0};
return invoke(query, 0).expand(res ->
this.handleResponse(res, query, iteration))
.flatMap(response -> Flux.fromIterable(response.collectItems()));
}
private Mono<ApiResponse> handleResponse(final ApiResponse response, final SearchQuery searchQuery, final int[] iteration) {
return hasNextPage(response) ? invoke(searchQuery, calculateOffset(++iteration[0])) : Mono.empty();
}
private Mono<ApiResponse> invoke(final SearchQuery query, final int offset) {
final var url = offset == 0 ? query.toUrlParams() : query.toUrlParamsWithOffset(offset);
return doInvoke(url).onErrorReturn(ApiResponse.emptyResponse());
}
private Mono<ApiResponse> doInvoke(final String endpoint) {
return webClient.get()
.uri(endpoint)
.retrieve()
.bodyToMono(ApiResponse.class);
}
And here is my service that is using this client
private final Map<String, Item> initialItems = new ConcurrentHashMap<>();
void work() {
final var executorService = Executors.newSingleThreadScheduledExecutor();
queryRepository.getSearchQueries().forEach(query -> {
reactiveClient.collectData(query).subscribe(item -> initialItems.put(item.getId(), item));
});
executorService.scheduleAtFixedRate(() -> {
if(isReady()) {
queryRepository.getSearchQueries().forEach(query -> {
reactiveClient.collectData(query).subscribe(this::process);
});
}
}, 0, 3, TimeUnit.SECONDS);
}
/**
* If after 2 second sleep size of initialItems remains the same,
* that most likely means that initial population phase is over,
* and we can proceed with further data processing
**/
private boolean isReady() {
try {
final var snapshotSize = initialItems.size();
Thread.sleep(2000);
return snapshotSize == initialItems.size();
} catch (Exception e) {
return false;
}
}
I think the code speaks for itself, I just want to finish first phase, which is initial data population and then start processing all incomming data.

Convert Observable of List to List of Observables and merge in RxJava

I'm learning Java with Android by creating Hacker News reader app.
What I'm trying to do is:
Send a request to /topstories, return Observable<List<int>>, emit when
request finishes.
Map each storyId to Observable<Story>
Merge Observables into one entity, which emits List<Story>, when all requests finishes.
And to the code:
private Observable<Story> getStoryById(int articleId) {
BehaviorSubject<Story> subject = BehaviorSubject.create();
// calls subject.onNext on success
JsonObjectRequest request = createStoryRequest(articleId, subject);
requestQueue.add(request);
return subject;
}
public Observable<ArrayList<Story>> getTopStories(int amount) {
Observable<ArrayList<Integer>> topStoryIds = (storyIdCache == null)
? fetchTopIds()
: Observable.just(storyIdCache);
return topStoryIds
.flatMap(id -> getStoryById(id))
// some magic here
}
Then we would use this like:
getTopStories(20)
.subscribe(stories -> ...)

You can try something like that
Observable<List<Integers>> ids = getIdsObservable();
Single<List<Story>> listSingle =
ids.flatMapIterable(ids -> ids)
.flatMap(id -> getStoryById(id)).toList();
Then you can subscribe to that Single to get the List<Story>

Please have a look at my solution. I changed your interface to return a Single for getStoryById(), because it should only return one value. After that, I created a for each Story a Single request and subscribed to all of them with Single.zip. Zip will execute given lambda, when all Singles are finished. On drawback is, that all requestes will be fired at once. If you do not want this, I will update my post. Please take into considerations that #elmorabea solution will also subscribe to the first 128 elements (BUFFER_SIZE = Math.max(1, Integer.getInteger("rx2.buffer-size", 128));), and to the next element when one finishes.
#Test
void name() {
Api api = mock(Api.class);
when(api.getTopStories()).thenReturn(Flowable.just(Arrays.asList(new Story(1), new Story(2))));
when(api.getStoryById(eq(1))).thenReturn(Single.just(new Story(888)));
when(api.getStoryById(eq(2))).thenReturn(Single.just(new Story(888)));
Flowable<List<Story>> listFlowable =
api.getTopStories()
.flatMapSingle(
stories -> {
List<Single<Story>> collect =
stories
.stream()
.map(story -> api.getStoryById(story.id))
.collect(Collectors.toList());
// possibly not the best idea to subscribe to all singles at the same time
Single<List<Story>> zip =
Single.zip(
collect,
objects -> {
return Arrays.stream(objects)
.map(o -> (Story) o)
.collect(Collectors.toList());
});
return zip;
});
TestSubscriber<List<Story>> listTestSubscriber =
listFlowable.test().assertComplete().assertValueCount(1).assertNoErrors();
List<List<Story>> values = listTestSubscriber.values();
List<Story> stories = values.get(0);
assertThat(stories.size()).isEqualTo(2);
assertThat(stories.get(0).id).isEqualTo(888);
assertThat(stories.get(1).id).isEqualTo(888);
}
interface Api {
Flowable<List<Story>> getTopStories();
Single<Story> getStoryById(int id);
}
static class Story {
private final int id;
Story(int id) {
this.id = id;
}
}

RxJava with vertx: can't have multiple subscriptions exception

I'm trying avoid vertx callback hell with RxJava.
But I have "rx.exceptions.OnErrorNotImplementedException: Cannot have multiple subscriptions". What's wrong here?
public class ShouldBeBetterSetter extends AbstractVerticle {
#Override
public void start(Future<Void> startFuture) throws Exception {
Func1<AsyncMap<String,Long>, Observable<Void>> obtainAndPutValueToMap = stringLongAsyncMap -> {
Long value = System.currentTimeMillis();
return stringLongAsyncMap.putObservable("timestamp", value)
.doOnError(Throwable::printStackTrace)
.doOnNext(aVoid -> System.out.println("succesfully putted"));
};
Observable<AsyncMap<String,Long>> clusteredMapObservable =
vertx.sharedData().<String,Long>getClusterWideMapObservable("mymap")
.doOnError(Throwable::printStackTrace);
vertx.periodicStream(3000).toObservable()
.flatMap(l-> clusteredMapObservable.flatMap(obtainAndPutValueToMap))
.forEach(o -> {
System.out.println("just printing.");
});
}
}
Working Verticle (without Rx) can be found here:
https://gist.github.com/IvanZelenskyy/9d50de8980b7bdf1e959e19593f7ce4a

vertx.sharedData().getClusterWideMapObservable("mymap") returns observable, which supports single subscriber only - hence exception. One solution worth a try is:
Observable<AsyncMap<String,Long>> clusteredMapObservable =
Observable.defer(
() -> vertx.sharedData().<String,Long>getClusterWideMapObservable("mymap")
);
That way every time clusteredMapObservable.flatMap() will be called, it will subscribe to new observable returned by Observable.defer().
EDIT
In case it's OK to use same AsyncMap, as pointed by #Ivan Zelenskyy, solution can be
Observable<AsyncMap<String,Long>> clusteredMapObservable =
vertx.sharedData().<String,Long>getClusterWideMapObservable("mymap").cache()

What's happening is that on each periodic emission, the foreach is re-subscribing to the clusteredMapObservable variable you defined above.
To fix, just move the call to vertx.sharedData().<String,Long>getClusterWideMapObservable("mymap") inside your periodic stream flatmap.
Something like this:
vertx.periodicStream(3000).toObservable()
.flatMap(l-> vertx.sharedData().<String,Long>getClusterWideMapObservable("mymap")
.doOnError(Throwable::printStackTrace)
.flatMap(obtainAndPutValueToMap))
.forEach(o -> {
System.out.println("just printing.");
});
UPDATE
If you don't like labmda in lambda, then don't. Here's an update without
vertx.periodicStream(3000).toObservable()
.flatMap(l-> {
return vertx.sharedData().<String,Long>getClusterWideMapObservable("mymap");
})
.doOnError(Throwable::printStackTrace)
.flatMap(obtainAndPutValueToMap)
.forEach(o -> {
System.out.println("just printing.");
});
PS - Your call to .flatMap(obtainAndPutValueToMap)) is also lambda in lambda - you've just moved it into a function.