Develop Reference

How to make parallel calls to the same service using spring Flux

I am working on spring reactive and need to call multiple calls sequentially to other REST API using webclient.
The issue is I am able to call multiple calls to other Rest API but response am not able to read without subscribe or block.
I can't use subscribe or block due to non reactive programming. Is there any way, i can merge while reading the response and send it as flux.
Below is the piece of code where I am stuck.
public Mono<DownloadDataLog> getDownload(Token dto, Mono<DataLogRequest> request) {
Mono<GraphQlCustomerResponse> profileResponse = customerProfileHandler.getMyUsageHomeMethods(dto, null);
DownloadDataLog responseObj = new DownloadDataLog();
ArrayList<Mono<List<dataUsageLogs>>> al = new ArrayList<>();
return Mono.zip(profileResponse, request).flatMap(tuple2 -> {
Flux<List<Mono<DataLogGqlRequest>>> userequest = prepareUserRequest(getListOfMdns(tuple2.getT1()),
tuple2.getT2());
Flux.from(userequest).flatMap(req -> {
for (Mono<DataLogGqlRequest> logReq : req) {
al.add(service.execute(logReq, dto));
}
responseObj.setAl(al);
return Mono.empty();
}).subscribe();
return Mono.just(responseObj);
});
}
private Mono<DataLogGqlRequest> prepareInnerRequest(Mono<DataLogGqlRequest> itemRequest, int v1,int v2){
return itemRequest.flatMap(req -> {
DataLogGqlRequest userRequest = new DataLogGqlRequest();
userRequest.setBillDate(req.getBillDate());
userRequest.setMdnNumber(req.getMdnNumber());
userRequest.setCposition(v1+"");
userRequest.setPposition(v2+"");
return Mono.just(userRequest);
});
}

How do I add a delay with data from the Mono?

I have a service that is returning a value which contains delay information.
public Mono<R> authenticate(
A authenticationRequest,
#RequestHeader Map<String, String> headers,
ServerHttpResponse serverHttpResponse) {
final AuthServiceResponse<R> authenticationResponse = authService.authenticate(authenticationRequest, headers);
serverHttpResponse.setStatusCode(authenticationResponse.getStatusCode());
return Mono.just(authenticationResponse.getOperationResponse())
.delayElement(authenticationResponse.getDelay());
}
I'd like to try to convert it so it is reactive I got this far...
public Mono<R> authenticate(
A authenticationRequest,
#RequestHeader Map<String, String> headers,
ServerHttpResponse serverHttpResponse) {
return authService.authenticate(authenticationRequest, headers)
.map(authenticationResponse->{
serverHttpResponse.setStatusCode(authenticationResponse.getStatusCode());
return authenticationResponse.getOperationResponse()
});
...
but I wasn't sure how to add the "delayElement" capability.

You can use Mono.fromCallable + delayElement within a flatMap like this:
return authService.authenticate(authenticationRequest, headers)
.flatMap(authenticationResponse -> {
return Mono.fromCallable(() -> authenticationResponse.getOperationResponse())
.delayElement(authenticationResponse.getDelay())
});
One thing to note... you cannot pass ServerHttpResponse in this situation as a parameter, but you have ServerWebExchange which has the request and response along with the headers. The complete solution is
public Mono<R> authenticate(
#RequestBody SimpleAuthenticationRequest authenticationRequest,
ServerWebExchange serverWebExchange) {
return authService
.authenticate(authenticationRequest, serverWebExchange.getRequest().getHeaders())
.doOnNext(
serviceResponse ->
serverWebExchange.getResponse().setStatusCode(serviceResponse.getStatusCode()))
.flatMap(
serviceResponse ->
Mono.fromCallable(serviceResponse::getOperationResponse)
.delayElement(serviceResponse.getDelay()));
}

Try this to add delay based on your authenticationResponse.getDelay() value
public Mono<Object> authenticate(Object authenticationRequest,#RequestHeader Object headers,
Object serverHttpResponse) {
return authenticate(authenticationRequest,headers)
.flatMap(authenticationResponse -> {
Mono<String> delayElement = Mono.just("add delay")
.delayElement(Duration.ofSeconds(authenticationResponse.getDelay()));
Mono<Object> actualResponse =Mono.just(authenticationResponse.getOperationResponse());
return Mono.zip(delayElement,actualResponse).map(tupleT2 -> tupleT2.getT2());
});
}
let me know if it doesn't work. i will try to find other way.

Reactive Spring Boot API wrapping Elasticsearch's async bulk indexing

I am developing prototype for a new project. The idea is to provide a Reactive Spring Boot microservice to bulk index documents in Elasticsearch. Elasticsearch provides a High Level Rest Client which provides an Async method to bulk process indexing requests. Async delivers callbacks using listeners are mentioned here. The callbacks receive index responses (per requests) in batches. I am trying to send this response back to the client as Flux. I have come up with something based on this blog post.
Controller
#RestController
public class AppController {
#SuppressWarnings("unchecked")
#RequestMapping(value = "/test3", method = RequestMethod.GET)
public Flux<String> index3() {
ElasticAdapter es = new ElasticAdapter();
JSONObject json = new JSONObject();
json.put("TestDoc", "Stack123");
Flux<String> fluxResponse = es.bulkIndex(json);
return fluxResponse;
}
ElasticAdapter
#Component
class ElasticAdapter {
String indexName = "test2";
private final RestHighLevelClient client;
private final ObjectMapper mapper;
private int processed = 1;
Flux<String> bulkIndex(JSONObject doc) {
return bulkIndexDoc(doc)
.doOnError(e -> System.out.print("Unable to index {}" + doc+ e));
}
private Flux<String> bulkIndexDoc(JSONObject doc) {
return Flux.create(sink -> {
try {
doBulkIndex(doc, bulkListenerToSink(sink));
} catch (JsonProcessingException e) {
sink.error(e);
}
});
}
private void doBulkIndex(JSONObject doc, BulkProcessor.Listener listener) throws JsonProcessingException {
System.out.println("Going to submit index request");
BiConsumer<BulkRequest, ActionListener<BulkResponse>> bulkConsumer =
(request, bulkListener) ->
client.bulkAsync(request, RequestOptions.DEFAULT, bulkListener);
BulkProcessor.Builder builder =
BulkProcessor.builder(bulkConsumer, listener);
builder.setBulkActions(10);
BulkProcessor bulkProcessor = builder.build();
// Submitting 5,000 index requests ( repeating same JSON)
for (int i = 0; i < 5000; i++) {
IndexRequest indexRequest = new IndexRequest(indexName, "person", i+1+"");
String json = doc.toJSONString();
indexRequest.source(json, XContentType.JSON);
bulkProcessor.add(indexRequest);
}
System.out.println("Submitted all docs
}
private BulkProcessor.Listener bulkListenerToSink(FluxSink<String> sink) {
return new BulkProcessor.Listener() {
#Override
public void beforeBulk(long executionId, BulkRequest request) {
}
#SuppressWarnings("unchecked")
#Override
public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
for (BulkItemResponse bulkItemResponse : response) {
JSONObject json = new JSONObject();
json.put("id", bulkItemResponse.getResponse().getId());
json.put("status", bulkItemResponse.getResponse().getResult
sink.next(json.toJSONString());
processed++;
}
if(processed >= 5000) {
sink.complete();
}
}
#Override
public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
failure.printStackTrace();
sink.error(failure);
}
};
}
public ElasticAdapter() {
// Logic to initialize Elasticsearch Rest Client
}
}
I used FluxSink to create the Flux of Responses to send back to the Client. At this point, I have no idea whether this correct or not.
My expectation is that the calling client should receive the responses in batches of 10 ( because bulk processor processess it in batches of 10 - builder.setBulkActions(10); ). I tried to consume the endpoint using Spring Webflix Client. But unable to work it out. This is what I tried
WebClient
public class FluxClient {
public static void main(String[] args) {
WebClient client = WebClient.create("http://localhost:8080");
Flux<String> responseFlux = client.get()
.uri("/test3")
.retrieve()
.bodyToFlux(String.class);
responseFlux.subscribe(System.out::println);
}
}
Nothing is printing on console as I expected. I tried to use System.out.println(responseFlux.blockFirst());. It prints all the responses as a single batch at the end and not in batches at .
If my approach is correct, what is the correct way to consume it? For the solution in my mind, this client will reside is another Webapp.
Notes: My understanding of Reactor API is limited. The version of elasticsearch used is 6.8.

So made the following changes to your code.
In ElasticAdapter,
public Flux<Object> bulkIndex(JSONObject doc) {
return bulkIndexDoc(doc)
.subscribeOn(Schedulers.elastic(), true)
.doOnError(e -> System.out.print("Unable to index {}" + doc+ e));
}
Invoked subscribeOn(Scheduler, requestOnSeparateThread) on the Flux, Got to know about it from, https://github.com/spring-projects/spring-framework/issues/21507
In FluxClient,
Flux<String> responseFlux = client.get()
.uri("/test3")
.headers(httpHeaders -> {
httpHeaders.set("Accept", "text/event-stream");
})
.retrieve()
.bodyToFlux(String.class);
responseFlux.delayElements(Duration.ofSeconds(1)).subscribe(System.out::println);
Added "Accept" header as "text/event-stream" and delayed Flux elements.
With the above changes, was able to get the response in real time from the server.

Spring WebFlux reactive WebSocket prevent connection closing

I'm working on simple chat module for my application using Spring WebFlux with ReactiveMongoRepository on backend and Angular 4 on front. I'm able to receive data through WebSocketSession but after streaming all messages from db i want to keep the connection so i could update message list. Can anyone give me clues how to achieve that, or maybe i'm following wrong assumptions ?
Java Backend responsible for WebSocket, my subscriber only logs current state, nothing relevant there:
WebFluxConfiguration:
#Configuration
#EnableWebFlux
public class WebSocketConfig {
private final WebSocketHandler webSocketHandler;
#Autowired
public WebSocketConfig(WebSocketHandler webSocketHandler) {
this.webSocketHandler = webSocketHandler;
}
#Bean
#Primary
public HandlerMapping webSocketMapping() {
Map<String, Object> map = new HashMap<>();
map.put("/websocket-messages", webSocketHandler);
SimpleUrlHandlerMapping mapping = new SimpleUrlHandlerMapping();
mapping.setOrder(10);
mapping.setUrlMap(map);
return mapping;
}
#Bean
public WebSocketHandlerAdapter handlerAdapter() {
return new WebSocketHandlerAdapter();
}
}
WebSocketHandler Implementation
#Component
public class MessageWebSocketHandler implements WebSocketHandler {
private final MessageRepository messageRepository;
private ObjectMapper mapper = new ObjectMapper();
private MessageSubscriber subscriber = new MessageSubscriber();
#Autowired
public MessageWebSocketHandler(MessageRepository messageRepository) {
this.messageRepository = messageRepository;
}
#Override
public Mono<Void> handle(WebSocketSession session) {
session.receive()
.map(WebSocketMessage::getPayloadAsText)
.map(this::toMessage)
.subscribe(subscriber::onNext, subscriber:: onError, subscriber::onComplete);
return session.send(
messageRepository.findAll()
.map(this::toJSON)
.map(session::textMessage));
}
private String toJSON(Message message) {
try {
return mapper.writeValueAsString(message);
} catch (JsonProcessingException e) {
throw new RuntimeException(e);
}
}
private Message toMessage(String json) {
try {
return mapper.readValue(json, Message.class);
} catch (IOException e) {
throw new RuntimeException("Invalid JSON:" + json, e);
}
}
}
and MongoRepo
#Repository
public interface MessageRepository extends
ReactiveMongoRepository<Message,String> {
}
FrontEnd Handling:
#Injectable()
export class WebSocketService {
private subject: Rx.Subject<MessageEvent>;
constructor() {
}
public connect(url): Rx.Subject<MessageEvent> {
if (!this.subject) {
this.subject = this.create(url);
console.log('Successfully connected: ' + url);
}
return this.subject;
}
private create(url): Rx.Subject<MessageEvent> {
const ws = new WebSocket(url);
const observable = Rx.Observable.create(
(obs: Rx.Observer<MessageEvent>) => {
ws.onmessage = obs.next.bind(obs);
ws.onerror = obs.error.bind(obs);
ws.onclose = obs.complete.bind(obs);
return ws.close.bind(ws);
});
const observer = {
next: (data: Object) => {
if (ws.readyState === WebSocket.OPEN) {
ws.send(JSON.stringify(data));
}
}
};
return Rx.Subject.create(observer, observable);
}
}
in other service i'm mapping observable from response to my type
constructor(private wsService: WebSocketService) {
this.messages = <Subject<MessageEntity>>this.wsService
.connect('ws://localhost:8081/websocket-messages')
.map((response: MessageEvent): MessageEntity => {
const data = JSON.parse(response.data);
return new MessageEntity(data.id, data.user_id, data.username, data.message, data.links);
});
}
and finally subscribtion with send function which i can't use because of closed connection:
ngOnInit() {
this.messages = [];
this._ws_subscription = this.chatService.messages.subscribe(
(message: MessageEntity) => {
this.messages.push(message);
},
error2 => {
console.log(error2.json());
},
() => {
console.log('Closed');
}
);
}
sendTestMessage() {
this.chatService.messages.next(new MessageEntity(null, '59ca30ac87e77d0f38237739', 'mickl', 'test message angular', null));
}

Assuming your chat messages are being persisted to the datastore as they're being received, you could use the tailable cursors feature in Spring Data MongoDB Reactive (see reference documentation).
So you could create a new method on your repository like:
public interface MessageRepository extends ReactiveSortingRepository< Message, String> {
#Tailable
Flux<Message> findWithTailableCursor();
}
Note that tailable cursors have some limitations: you mongo collection needs to be capped and entries are streamed in their order of insertion.
Spring WebFlux websocket support does not yet support STOMP nor message brokers, but this might be a better choice for such a use case.

RxJava onErrorResumeNext()

I have two observables (named A and B for simplicity) and one subscriber. So, the Subscriber subscribes to A and if there's an error on A then B (which is the fallback) kicks in. Now, whenever A hits an error B gets called fine, however A calls onComplete() on the subscriber, so B response never reaches the subscriber even if B execution is successful.
Is this the normal behaviour? I thought onErrorResumeNext() should continue the stream and notify the subscriber once completed as noted in the documentation (https://github.com/ReactiveX/RxJava/wiki/Error-Handling-Operators#onerrorresumenext).
This is the overall structure of what I'm doing (omitted several "boring" code):
public Observable<ModelA> observeGetAPI(){
return retrofitAPI.getObservableAPI1()
.flatMap(observableApi1Response -> {
ModelA model = new ModelA();
model.setApi1Response(observableApi1Response);
return retrofitAPI.getObservableAPI2()
.map(observableApi2Response -> {
// Blah blah blah...
return model;
})
.onErrorResumeNext(observeGetAPIFallback(model))
.subscribeOn(Schedulers.newThread())
})
.onErrorReturn(throwable -> {
// Blah blah blah...
return model;
})
.subscribeOn(Schedulers.newThread());
}
private Observable<ModelA> observeGetAPIFallback(ModelA model){
return retrofitAPI.getObservableAPI3().map(observableApi3Response -> {
// Blah blah blah...
return model;
}).onErrorReturn(throwable -> {
// Blah blah blah...
return model;
})
.subscribeOn(Schedulers.immediate());
}
Subscription subscription;
subscription = observeGetAPI.subscribe(ModelA -> {
// IF THERE'S AN ERROR WE NEVER GET B RESPONSE HERE...
}, throwable ->{
// WE NEVER GET HERE... onErrorResumeNext()
},
() -> { // IN CASE OF AN ERROR WE GET STRAIGHT HERE, MEANWHILE, B GETS EXECUTED }
);
Any ideas what I'm doing wrong?
Thanks!
EDIT:
Here's a rough timeline of what's happening:
---> HTTP GET REQUEST B
<--- HTTP 200 REQUEST B RESPONSE (SUCCESS)
---> HTTP GET REQUEST A
<--- HTTP 200 REQUEST A RESPONSE (FAILURE!)
---> HTTP GET FALLBACK A
** onComplete() called! ---> Subscriber never gets fallback response since onComplete() gets called before time.
<--- HTTP 200 FALLBACK A RESPONSE (SUCCESS)
And here's a link to a simple diagram I made which represent's what I want to happen:
Diagram

The Rx calls used in the following should simulate what you are doing with Retrofit.
fallbackObservable =
Observable
.create(new Observable.OnSubscribe<String>() {
#Override
public void call(Subscriber<? super String> subscriber) {
logger.v("emitting A Fallback");
subscriber.onNext("A Fallback");
subscriber.onCompleted();
}
})
.delay(1, TimeUnit.SECONDS)
.onErrorReturn(new Func1<Throwable, String>() {
#Override
public String call(Throwable throwable) {
logger.v("emitting Fallback Error");
return "Fallback Error";
}
})
.subscribeOn(Schedulers.immediate());
stringObservable =
Observable
.create(new Observable.OnSubscribe<String>() {
#Override
public void call(Subscriber<? super String> subscriber) {
logger.v("emitting B");
subscriber.onNext("B");
subscriber.onCompleted();
}
})
.delay(1, TimeUnit.SECONDS)
.flatMap(new Func1<String, Observable<String>>() {
#Override
public Observable<String> call(String s) {
logger.v("flatMapping B");
return Observable
.create(new Observable.OnSubscribe<String>() {
#Override
public void call(Subscriber<? super String> subscriber) {
logger.v("emitting A");
subscriber.onNext("A");
subscriber.onCompleted();
}
})
.delay(1, TimeUnit.SECONDS)
.map(new Func1<String, String>() {
#Override
public String call(String s) {
logger.v("A completes but contains invalid data - throwing error");
throw new NotImplementedException("YUCK!");
}
})
.onErrorResumeNext(fallbackObservable)
.subscribeOn(Schedulers.newThread());
}
})
.onErrorReturn(new Func1<Throwable, String>() {
#Override
public String call(Throwable throwable) {
logger.v("emitting Return Error");
return "Return Error";
}
})
.subscribeOn(Schedulers.newThread());
subscription = stringObservable.subscribe(
new Action1<String>() {
#Override
public void call(String s) {
logger.v("onNext " + s);
}
},
new Action1<Throwable>() {
#Override
public void call(Throwable throwable) {
logger.v("onError");
}
},
new Action0() {
#Override
public void call() {
logger.v("onCompleted");
}
});
The output from the log statements is:
RxNewThreadScheduler-1 emitting B
RxComputationThreadPool-1 flatMapping B
RxNewThreadScheduler-2 emitting A
RxComputationThreadPool-2 A completes but contains invalid data - throwing error
RxComputationThreadPool-2 emitting A Fallback
RxComputationThreadPool-1 onNext A Fallback
RxComputationThreadPool-1 onCompleted
This seems like what you are looking for but maybe I'm missing something.