New to reactive programming and also Spring Webflux, I have a method to get value from redis and expire the key under certain conditions. But the code expire key always not work.
my current implmention:
private Mono<MyObject> getMyObjectFromCache(String url) {
RMapReactive<String, String> rMap = redissonReactiveClient.getMap(url);
return rMap.readAllMap()
.flatMap(m ->
rMap.remainTimeToLive()
.flatMap(ttl -> {
final long renewalThreshold = 60 * 60 * 1000;
if (ttl <= renewalThreshold) {
System.out.println("start expiring");
// it doesn't work without subscribe()
rMap.expire(2, TimeUnit.HOURS);
}
return Mono.just(JSONObject.parseObject(JSON.toJSONString(m), MyObject.class));
}
)
);
}
expire method returns Mono<Boolean>
public Mono<MyObject> getMyObjInfo(String url) {
// something else
return getMyObjectFromFromCache(url).switchIfEmpty(Mono.defer(() -> getMyObjectFromRemoteService(url)));
}
CustomGatewayFilter
#Override
public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
ServerHttpRequest request = exchange.getRequest();
ServerHttpResponse response = exchange.getResponse();
URI uri = request.getURI();
return getMyObjInfo(uri.getPath())
.flatMap(api -> {
// something else
return chain.filter(exchange.mutate().request(request).build());
});
when I test the filter , only print start expiring , but it doesn't work.
if i add subscribe or block, it can work. obviously this is not a good idea, I shouldn't break the reactor chain.
please could I have a correctly way to write this?
thanks
In reactive you need to combine all async operations into a flow, chaining publishers (Mono/Flux) using various reactive operators (assembly time) and then subscribe to it (subscription time). You are right that calling subscribe explicitly is a bad practice and should be avoided. Spring WebFlux subscribes to the provided flow behind the scene.
In your code you are breaking the flow by not chaining rMap.expire(2, TimeUnit.HOURS);. You could rewrite the code like this
private Mono<MyObject> getMyObjectFromCache(String url) {
RMapReactive<String, String> rMap = redissonReactiveClient.getMap(url);
return rMap.readAllMap()
.flatMap(m ->
rMap.remainTimeToLive()
.flatMap(ttl -> {
final long renewalThreshold = 60 * 60 * 1000;
if (ttl <= renewalThreshold) {
System.out.println("start expiring");
// it doesn't work without subscribe()
return rMap.expire(2, TimeUnit.HOURS);
}
return Mono.just(false);
})
.then(JSONObject.parseObject(JSON.toJSONString(m), MyObject.class))
);
}
Related
I'm writing a web client for an API.
Every request should be accompanied with an access_token that may overdue.
What i want is to catch cases where request is failed due to overdue token, refresh it and retry the request. The issue is that I'm receiving the token as a Mono via the same webClient.
What I actually want is something like this:
private AtomicReference<Token> token
...
public Mono<ApiResponse> callApi() {
return Mono.justOrEmpty(token.get())
.switchIfEmpty(
Mono.defer(() -> auth()
.doOnNext(token::set)))
.flatMap(token -> performRequest(token))
.doOnError(e -> {
var newToken = auth().block() // I obviously can't block here
token.set(newToken);
})
.retry(5);
}
private Mono<Token> auth() {
// api call here that returns token
}
So what's the correct reactive way to update token and then retry request with it?
===UPD===
I managed to handle it, however, it looks a bit wanky.
Probably you have a better solution?
private AtomicReference<TokenHolder> tokenHolder = new AtomicReference<>();
private AtomicBoolean lastQueryFailed = new AtomicBoolean();
public Mono<ApiResponse> getApiResponse() {
return Mono.defer(this::getToken)
.flatMap(this::requestApi)
.doOnError((e) -> {
log.error(e)
lastQueryFailed.set(true);
})
.retryWhen(Retry.backoff(
3,
Duration.ofSeconds(2)
));
}
private Mono<Token> getToken() {
if (tokenHolder.get() == null) {
return auth()
.doOnNext(tokenHolder::set);
}
if (!lastQueryFailed.get()) {
return Mono.just(tokenHolder.get());
}
return auth()
.doOnNext(tokenHolder::set);
}
private Mono<Token> auth() {
// api call here that returns token
}
Mono.onErrorResume seems to be the operator you are looking for: It allows you to switch to a different Mono (in your case one based on auth()) when an error occurs:
public Mono<ApiResponse> getApiResponse() {
return Mono.justOrEmpty(token.get())
.switchIfEmpty(Mono.defer(() -> auth()
.doOnNext(token::set)))
.flatMap(this::requestApi)
.onErrorResume(SecurityException.class, error -> auth()
.doOnNext(token::set)
.flatMap(this::requestApi))
.retryWhen(Retry.backoff(3, Duration.ofSeconds(2)));
}
Note that I assumed you get a SecurityException if the token is overdue. You can change it to a different class, or even a Predicate<Throwable> to catch the overdue token exception. It is recommended to catch this specific error instead of all errors, else it will also refresh the token on other errors, like when the service is unreachable.
I am working on spring reactive and need to call multiple calls sequentially to other REST API using webclient.
The issue is I am able to call multiple calls to other Rest API but response am not able to read without subscribe or block.
I can't use subscribe or block due to non reactive programming. Is there any way, i can merge while reading the response and send it as flux.
Below is the piece of code where I am stuck.
public Mono<DownloadDataLog> getDownload(Token dto, Mono<DataLogRequest> request) {
Mono<GraphQlCustomerResponse> profileResponse = customerProfileHandler.getMyUsageHomeMethods(dto, null);
DownloadDataLog responseObj = new DownloadDataLog();
ArrayList<Mono<List<dataUsageLogs>>> al = new ArrayList<>();
return Mono.zip(profileResponse, request).flatMap(tuple2 -> {
Flux<List<Mono<DataLogGqlRequest>>> userequest = prepareUserRequest(getListOfMdns(tuple2.getT1()),
tuple2.getT2());
Flux.from(userequest).flatMap(req -> {
for (Mono<DataLogGqlRequest> logReq : req) {
al.add(service.execute(logReq, dto));
}
responseObj.setAl(al);
return Mono.empty();
}).subscribe();
return Mono.just(responseObj);
});
}
private Mono<DataLogGqlRequest> prepareInnerRequest(Mono<DataLogGqlRequest> itemRequest, int v1,int v2){
return itemRequest.flatMap(req -> {
DataLogGqlRequest userRequest = new DataLogGqlRequest();
userRequest.setBillDate(req.getBillDate());
userRequest.setMdnNumber(req.getMdnNumber());
userRequest.setCposition(v1+"");
userRequest.setPposition(v2+"");
return Mono.just(userRequest);
});
}
I am developing prototype for a new project. The idea is to provide a Reactive Spring Boot microservice to bulk index documents in Elasticsearch. Elasticsearch provides a High Level Rest Client which provides an Async method to bulk process indexing requests. Async delivers callbacks using listeners are mentioned here. The callbacks receive index responses (per requests) in batches. I am trying to send this response back to the client as Flux. I have come up with something based on this blog post.
Controller
#RestController
public class AppController {
#SuppressWarnings("unchecked")
#RequestMapping(value = "/test3", method = RequestMethod.GET)
public Flux<String> index3() {
ElasticAdapter es = new ElasticAdapter();
JSONObject json = new JSONObject();
json.put("TestDoc", "Stack123");
Flux<String> fluxResponse = es.bulkIndex(json);
return fluxResponse;
}
ElasticAdapter
#Component
class ElasticAdapter {
String indexName = "test2";
private final RestHighLevelClient client;
private final ObjectMapper mapper;
private int processed = 1;
Flux<String> bulkIndex(JSONObject doc) {
return bulkIndexDoc(doc)
.doOnError(e -> System.out.print("Unable to index {}" + doc+ e));
}
private Flux<String> bulkIndexDoc(JSONObject doc) {
return Flux.create(sink -> {
try {
doBulkIndex(doc, bulkListenerToSink(sink));
} catch (JsonProcessingException e) {
sink.error(e);
}
});
}
private void doBulkIndex(JSONObject doc, BulkProcessor.Listener listener) throws JsonProcessingException {
System.out.println("Going to submit index request");
BiConsumer<BulkRequest, ActionListener<BulkResponse>> bulkConsumer =
(request, bulkListener) ->
client.bulkAsync(request, RequestOptions.DEFAULT, bulkListener);
BulkProcessor.Builder builder =
BulkProcessor.builder(bulkConsumer, listener);
builder.setBulkActions(10);
BulkProcessor bulkProcessor = builder.build();
// Submitting 5,000 index requests ( repeating same JSON)
for (int i = 0; i < 5000; i++) {
IndexRequest indexRequest = new IndexRequest(indexName, "person", i+1+"");
String json = doc.toJSONString();
indexRequest.source(json, XContentType.JSON);
bulkProcessor.add(indexRequest);
}
System.out.println("Submitted all docs
}
private BulkProcessor.Listener bulkListenerToSink(FluxSink<String> sink) {
return new BulkProcessor.Listener() {
#Override
public void beforeBulk(long executionId, BulkRequest request) {
}
#SuppressWarnings("unchecked")
#Override
public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
for (BulkItemResponse bulkItemResponse : response) {
JSONObject json = new JSONObject();
json.put("id", bulkItemResponse.getResponse().getId());
json.put("status", bulkItemResponse.getResponse().getResult
sink.next(json.toJSONString());
processed++;
}
if(processed >= 5000) {
sink.complete();
}
}
#Override
public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
failure.printStackTrace();
sink.error(failure);
}
};
}
public ElasticAdapter() {
// Logic to initialize Elasticsearch Rest Client
}
}
I used FluxSink to create the Flux of Responses to send back to the Client. At this point, I have no idea whether this correct or not.
My expectation is that the calling client should receive the responses in batches of 10 ( because bulk processor processess it in batches of 10 - builder.setBulkActions(10); ). I tried to consume the endpoint using Spring Webflix Client. But unable to work it out. This is what I tried
WebClient
public class FluxClient {
public static void main(String[] args) {
WebClient client = WebClient.create("http://localhost:8080");
Flux<String> responseFlux = client.get()
.uri("/test3")
.retrieve()
.bodyToFlux(String.class);
responseFlux.subscribe(System.out::println);
}
}
Nothing is printing on console as I expected. I tried to use System.out.println(responseFlux.blockFirst());. It prints all the responses as a single batch at the end and not in batches at .
If my approach is correct, what is the correct way to consume it? For the solution in my mind, this client will reside is another Webapp.
Notes: My understanding of Reactor API is limited. The version of elasticsearch used is 6.8.
So made the following changes to your code.
In ElasticAdapter,
public Flux<Object> bulkIndex(JSONObject doc) {
return bulkIndexDoc(doc)
.subscribeOn(Schedulers.elastic(), true)
.doOnError(e -> System.out.print("Unable to index {}" + doc+ e));
}
Invoked subscribeOn(Scheduler, requestOnSeparateThread) on the Flux, Got to know about it from, https://github.com/spring-projects/spring-framework/issues/21507
In FluxClient,
Flux<String> responseFlux = client.get()
.uri("/test3")
.headers(httpHeaders -> {
httpHeaders.set("Accept", "text/event-stream");
})
.retrieve()
.bodyToFlux(String.class);
responseFlux.delayElements(Duration.ofSeconds(1)).subscribe(System.out::println);
Added "Accept" header as "text/event-stream" and delayed Flux elements.
With the above changes, was able to get the response in real time from the server.
I have an Play application that handles WebSocket requests. The routes file contains this line:
GET /testsocket controllers.HomeController.defaultRoomSocket
An already working, synchronous version looks like this: (adapted from 2.7.x docs)
public WebSocket defaultRoomSocket() {
return WebSocket.Text.accept(
request -> ActorFlow.actorRef(MyWebSocketActor::props, actorSystem, materializer));
}
As stated in https://www.playframework.com/documentation/2.7.x/JavaWebSockets#Accepting-a-WebSocket-asynchronously I changed the signature to
public CompletionStage<WebSocket> defaultRoomSocket(){
//returning a CompletionStage here, using the "ask pattern"
//to get the needed Flow from an other Actor
}
From here I run into the following problem:
Cannot use a method returning java.util.concurrent.CompletionStage[play.mvc.WebSocket] as a Handler for requests
Further more, 'WebSocket' has no TypeParameter, as the documentation suggests. What is the appropriate way to accept a WebSocket request async?
The documentation indeed need to be updated, I think some bits were missed in the refactoring of the websockets in #5055.
To get async processing, you should use the acceptOrResult method that takes a CompletionStage as return type instead of a flow. This can then return either a Result or an Akka Flow, using a functional programming helper (F.Either). In fact, here's how the accept method is implemented:
public WebSocket accept(Function<Http.RequestHeader, Flow<In, Out, ?>> f) {
return acceptOrResult(
request -> CompletableFuture.completedFuture(F.Either.Right(f.apply(request))));
}
As you can see, all it does is call the async version with a completedFuture.
To fully make it async and get to what I think you're trying to achieve, you'd do something like this:
public WebSocket ws() {
return WebSocket.Json.acceptOrResult(request -> {
if (sameOriginCheck(request)) {
final CompletionStage<Flow<JsonNode, JsonNode, NotUsed>> future = wsFutureFlow(request);
final CompletionStage<Either<Result, Flow<JsonNode, JsonNode, ?>>> stage = future.thenApply(Either::Right);
return stage.exceptionally(this::logException);
} else {
return forbiddenResult();
}
});
}
#SuppressWarnings("unchecked")
private CompletionStage<Flow<JsonNode, JsonNode, NotUsed>> wsFutureFlow(Http.RequestHeader request) {
long id = request.asScala().id();
UserParentActor.Create create = new UserParentActor.Create(Long.toString(id));
return ask(userParentActor, create, t).thenApply((Object flow) -> {
final Flow<JsonNode, JsonNode, NotUsed> f = (Flow<JsonNode, JsonNode, NotUsed>) flow;
return f.named("websocket");
});
}
private CompletionStage<Either<Result, Flow<JsonNode, JsonNode, ?>>> forbiddenResult() {
final Result forbidden = Results.forbidden("forbidden");
final Either<Result, Flow<JsonNode, JsonNode, ?>> left = Either.Left(forbidden);
return CompletableFuture.completedFuture(left);
}
private Either<Result, Flow<JsonNode, JsonNode, ?>> logException(Throwable throwable) {
logger.error("Cannot create websocket", throwable);
Result result = Results.internalServerError("error");
return Either.Left(result);
}
(this is taken from the play-java-websocket-example, which might be of interest)
As you can see, it first goes through a few stages before returning either a websocket connection or a HTTP status.
How to end chained requests in Rx Vert.X ?
HttpClient client = Vertx.vertx().createHttpClient();
HttpClientRequest request = client.request(HttpMethod.POST,
"someURL")
.putHeader("content-type", "application/x-www-form-urlencoded")
.putHeader("content-length", Integer.toString(jsonData.length())).write(jsonData);
request.toObservable().
//flatmap HttpClientResponse -> Observable<Buffer>
flatMap(httpClientResponse -> { //something
return httpClientResponse.toObservable();
}).
map(buffer -> {return buffer.toString()}).
//flatmap data -> Observable<HttpClientResponse>
flatMap(postData -> client.request(HttpMethod.POST,
someURL")
.putHeader("content-type", "application/x-www-form-urlencoded")
.putHeader("content-length", Integer.toString(postData.length())).write(postData).toObservable()).
//flatmap HttpClientResponse -> Observable<Buffer>
flatMap(httpClientResponse -> {
return httpClientResponse.toObservable();
})......//other operators
request.end();
Notice that I have .end() for the top request. How do I end request that is inside of the .flatmap ? Do I even need to end it ?
There are multiple ways to ensure to call request.end(). But I would dig into documentation of Vert.x or just open source code if there is one, to see if it does call end() for you. Otherwise one could be
final HttpClientRequest request = ...
request.toObservable()
.doOnUnsubscribe(new Action0() {
#Override
public void call() {
request.end();
}
});
I think you can do something like the following code.
The main idea is that you don't directly use the HttpClientRequest as obtained by the Vertx client. Instead you create another flowable that will invoke end() as soon as the first subscription is received.
Here, for instance, you can obtain the request through a pair custom methods: in this case request1() and request2(). They both use doOnSubscribe() to trigger the end() you need. Read its description on the ReactiveX page.
This examle uses vertx and reactivex, I hope you could use this set up.
import io.reactivex.Flowable;
import io.vertx.core.http.HttpMethod;
import io.vertx.reactivex.core.Vertx;
import io.vertx.reactivex.core.buffer.Buffer;
import io.vertx.reactivex.core.http.HttpClient;
import io.vertx.reactivex.core.http.HttpClientRequest;
import io.vertx.reactivex.core.http.HttpClientResponse;
import org.junit.Test;
public class StackOverflow {
#Test public void test(){
Buffer jsonData = Buffer.buffer("..."); // the json data.
HttpClient client = Vertx.vertx().createHttpClient(); // the vertx client.
request1(client)
.flatMap(httpClientResponse -> httpClientResponse.toFlowable())
.map(buffer -> buffer.toString())
.flatMap(postData -> request2(client, postData) )
.forEach( httpResponse -> {
// do something with returned data);
});
}
private Flowable<HttpClientResponse> request1(HttpClient client) {
HttpClientRequest request = client.request(HttpMethod.POST,"someURL");
return request
.toFlowable()
.doOnSubscribe( subscription -> request.end() );
}
private Flowable<HttpClientResponse> request2(HttpClient client, String postData) {
HttpClientRequest request = client.request(HttpMethod.POST,"someURL");
// do something with postData
return request
.toFlowable()
.doOnSubscribe( subscription -> request.end() );
}
}