How to end chained requests in Rx Vert.X ?
HttpClient client = Vertx.vertx().createHttpClient();
HttpClientRequest request = client.request(HttpMethod.POST,
"someURL")
.putHeader("content-type", "application/x-www-form-urlencoded")
.putHeader("content-length", Integer.toString(jsonData.length())).write(jsonData);
request.toObservable().
//flatmap HttpClientResponse -> Observable<Buffer>
flatMap(httpClientResponse -> { //something
return httpClientResponse.toObservable();
}).
map(buffer -> {return buffer.toString()}).
//flatmap data -> Observable<HttpClientResponse>
flatMap(postData -> client.request(HttpMethod.POST,
someURL")
.putHeader("content-type", "application/x-www-form-urlencoded")
.putHeader("content-length", Integer.toString(postData.length())).write(postData).toObservable()).
//flatmap HttpClientResponse -> Observable<Buffer>
flatMap(httpClientResponse -> {
return httpClientResponse.toObservable();
})......//other operators
request.end();
Notice that I have .end() for the top request. How do I end request that is inside of the .flatmap ? Do I even need to end it ?
There are multiple ways to ensure to call request.end(). But I would dig into documentation of Vert.x or just open source code if there is one, to see if it does call end() for you. Otherwise one could be
final HttpClientRequest request = ...
request.toObservable()
.doOnUnsubscribe(new Action0() {
#Override
public void call() {
request.end();
}
});
I think you can do something like the following code.
The main idea is that you don't directly use the HttpClientRequest as obtained by the Vertx client. Instead you create another flowable that will invoke end() as soon as the first subscription is received.
Here, for instance, you can obtain the request through a pair custom methods: in this case request1() and request2(). They both use doOnSubscribe() to trigger the end() you need. Read its description on the ReactiveX page.
This examle uses vertx and reactivex, I hope you could use this set up.
import io.reactivex.Flowable;
import io.vertx.core.http.HttpMethod;
import io.vertx.reactivex.core.Vertx;
import io.vertx.reactivex.core.buffer.Buffer;
import io.vertx.reactivex.core.http.HttpClient;
import io.vertx.reactivex.core.http.HttpClientRequest;
import io.vertx.reactivex.core.http.HttpClientResponse;
import org.junit.Test;
public class StackOverflow {
#Test public void test(){
Buffer jsonData = Buffer.buffer("..."); // the json data.
HttpClient client = Vertx.vertx().createHttpClient(); // the vertx client.
request1(client)
.flatMap(httpClientResponse -> httpClientResponse.toFlowable())
.map(buffer -> buffer.toString())
.flatMap(postData -> request2(client, postData) )
.forEach( httpResponse -> {
// do something with returned data);
});
}
private Flowable<HttpClientResponse> request1(HttpClient client) {
HttpClientRequest request = client.request(HttpMethod.POST,"someURL");
return request
.toFlowable()
.doOnSubscribe( subscription -> request.end() );
}
private Flowable<HttpClientResponse> request2(HttpClient client, String postData) {
HttpClientRequest request = client.request(HttpMethod.POST,"someURL");
// do something with postData
return request
.toFlowable()
.doOnSubscribe( subscription -> request.end() );
}
}
Related
New to reactive programming and also Spring Webflux, I have a method to get value from redis and expire the key under certain conditions. But the code expire key always not work.
my current implmention:
private Mono<MyObject> getMyObjectFromCache(String url) {
RMapReactive<String, String> rMap = redissonReactiveClient.getMap(url);
return rMap.readAllMap()
.flatMap(m ->
rMap.remainTimeToLive()
.flatMap(ttl -> {
final long renewalThreshold = 60 * 60 * 1000;
if (ttl <= renewalThreshold) {
System.out.println("start expiring");
// it doesn't work without subscribe()
rMap.expire(2, TimeUnit.HOURS);
}
return Mono.just(JSONObject.parseObject(JSON.toJSONString(m), MyObject.class));
}
)
);
}
expire method returns Mono<Boolean>
public Mono<MyObject> getMyObjInfo(String url) {
// something else
return getMyObjectFromFromCache(url).switchIfEmpty(Mono.defer(() -> getMyObjectFromRemoteService(url)));
}
CustomGatewayFilter
#Override
public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
ServerHttpRequest request = exchange.getRequest();
ServerHttpResponse response = exchange.getResponse();
URI uri = request.getURI();
return getMyObjInfo(uri.getPath())
.flatMap(api -> {
// something else
return chain.filter(exchange.mutate().request(request).build());
});
when I test the filter , only print start expiring , but it doesn't work.
if i add subscribe or block, it can work. obviously this is not a good idea, I shouldn't break the reactor chain.
please could I have a correctly way to write this?
thanks
In reactive you need to combine all async operations into a flow, chaining publishers (Mono/Flux) using various reactive operators (assembly time) and then subscribe to it (subscription time). You are right that calling subscribe explicitly is a bad practice and should be avoided. Spring WebFlux subscribes to the provided flow behind the scene.
In your code you are breaking the flow by not chaining rMap.expire(2, TimeUnit.HOURS);. You could rewrite the code like this
private Mono<MyObject> getMyObjectFromCache(String url) {
RMapReactive<String, String> rMap = redissonReactiveClient.getMap(url);
return rMap.readAllMap()
.flatMap(m ->
rMap.remainTimeToLive()
.flatMap(ttl -> {
final long renewalThreshold = 60 * 60 * 1000;
if (ttl <= renewalThreshold) {
System.out.println("start expiring");
// it doesn't work without subscribe()
return rMap.expire(2, TimeUnit.HOURS);
}
return Mono.just(false);
})
.then(JSONObject.parseObject(JSON.toJSONString(m), MyObject.class))
);
}
The issue I need to solve (in Java) is:
Fetch an array of games from an API.
Iterate over games N times:
(async / concurrent)
Make an individual HTTP request for every game in order to get its
details.
Store the details of it in a gamesWithDetails array.
Done. I have my array gamesWithDetails.
I cannot fetch the details of all the games with a single request, I have to hit the API endpoint every time per game. So I want to execute these requests asynchronously from each other.
This is a working example in JavaScript in case it's useful. However I'd like to make it work for Spring Boot.
axios.get(`https://la2.api.riotgames.com/lol/match/v4/matchlists/by-account/${data.accountId}`, {
headers: { "X-Riot-Token": "asdasdasdasdadasdasdasd"}
})
.then(resp => {
const promises = [];
for ( match of resp.data.matches ) {
promises.push(
axios.get(`https://la2.api.riotgames.com/lol/match/v4/matches/${match.gameId}`, {
headers: { "X-Riot-Token": "asdasdasdasdasdasdasdasd"}
})
)
}
Promise.all(promises)
.then(matchesDetails => {
matchesDetails.forEach(({ data }) => console.log(data.gameId));
});
})
Basically you will want to do something like this:
package com.example.demo;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.client.RestTemplate;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executor;
import java.util.concurrent.Executors;
import java.util.stream.Collectors;
public class GamesProcessor {
private static final String GAME_URI_BASE = "https://la2.api.riotgames.com/lol/match/v4/matches/";
private static final String ACCOUNT_URI_BASE = "https://la2.api.riotgames.com/lol/match/v4/matchlists/by-account/";
private Executor executor = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors() - 1);
#Autowired
private RestTemplate restTemplate;
public void processGames(String accountId) throws JsonProcessingException, ExecutionException, InterruptedException {
String responseAsString = restTemplate.getForObject(ACCOUNT_URI_BASE + accountId, String.class);
ObjectMapper objectMapper = new ObjectMapper();
if (responseAsString != null) {
Map<String, Object> response = objectMapper.readValue(responseAsString, new TypeReference<Map<String, Object>>() {
});
List<Map<String, Object>> matches = (List<Map<String, Object>>) ((Map<String, Object>) response.get("data")).get("matches");
List<CompletableFuture<Void>> futures = matches.stream()
.map(m -> (String) m.get("gameId"))
.map(gameId -> CompletableFuture.supplyAsync(() -> restTemplate.getForObject(GAME_URI_BASE + gameId, String.class), executor)
.thenAccept(r -> {
System.out.println(r); //do whatever you wish with the response here
}))
.collect(Collectors.toList());
// now we execute all requests asynchronously
CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).get();
}
}
}
Please note that it is not a refined code, but just a quick example of how to achieve this. Ideally you would replace that JSON processing that I've done "by hand" using Map by a response bean that matches the structure of the response you get from the service you are calling.
A quick walk through:
String responseAsString = restTemplate.getForObject(ACCOUNT_URI_BASE + accountId, String.class);
This executes the first REST request and gets it as a String (the JSON response). You will want to properly map this using a Bean object instead. Then this is processed using the ObjectMapper provided by Jackson and transformed into a map so you can navigate the JSON and get the matches.
List<CompletableFuture<Void>> futures = matches.stream()
.map(m -> (String) m.get("gameId"))
.map(gameId -> CompletableFuture.supplyAsync(() -> restTemplate.getForObject(GAME_URI_BASE + gameId, String.class), executor)
.thenAccept(r -> {
System.out.println(r); //do whatever you wish with the response here
}))
.collect(Collectors.toList());
Once we have all the matches we will use the Stream API to transform them into CompletableFutures that will be executed asynchronously. Each thread will make another request in order to get the response for each individual matchId.
System.out.println(r);
This will be executed for each response that you get for each matchId, just like in your example. This should also be replaced by a proper bean matching the output for clearer processing.
Note that List<CompletableFuture<Void>> futures only "holds the code" but will not get executed until we combine everything in the end using CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).get(); and execute the blocking get() method.
Quite interesting question as JavaScript implements the famous event loop which means its functions are asynchronous and non-blocking. Spring Boot restTemplate class will block the thread of execution until the response is back, therefore wasting a lot of resources (the one thread per request model).
#Slacky's answer is technically right as you asked about asynchronous HTTP requests but I'd like to share a better option which is both asynchronous and non-blocking, meaning a single thread is able to handle 100s or even 1000s of requests and their responses (reactive programming).
The way to implement in Spring Boot the equivalent to your JavaScript example is to use the Project Reactor WebClient class which is a non-blocking, reactive client to perform HTTP requests.
It is also worth mentioning that Java being statically typed requires you to declare classes to represent your data, in this case something like (using Lombok for brevity):
#Data
class Match {
private String gameId;
// ...
}
#Data
class MatchDetails {
// ...
}
Here is the code following #Slacky's answer naming convention to make the comparison easier.
public class GamesProcessor {
private static final String BASE_URL = "https://la2.api.riotgames.com";
private static final String GAME_URI = "/lol/match/v4/matches/%s";
private static final String ACCOUNT_URI = "/lol/match/v4/matchlists/by-account/%s";
public static List<MatchDetails> processGames(String accountId) {
final WebClient webClient = WebClient
.builder()
.baseUrl(BASE_URL)
.defaultHeader("X-Riot-Token", "asdasdasdasdadasdasdasd")
.build();
// Issues the first request to get list of matches
List<Match> matches = webClient
.get()
.uri(String.format(ACCOUNT_URI, accountId))
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToMono(new ParameterizedTypeReference<List<Match>>() {})
.block(); // blocks to wait for response
// Processes the list of matches asynchronously and collect all responses in a list of matches details
return Flux.fromIterable(matches)
.flatMap(match -> webClient
.get()
.uri(String.format(GAME_URI, match.getGameId()))
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToMono(MatchDetails.class))
.collectList()
.block(); // Blocks to wait for all responses
}
}
I am developing prototype for a new project. The idea is to provide a Reactive Spring Boot microservice to bulk index documents in Elasticsearch. Elasticsearch provides a High Level Rest Client which provides an Async method to bulk process indexing requests. Async delivers callbacks using listeners are mentioned here. The callbacks receive index responses (per requests) in batches. I am trying to send this response back to the client as Flux. I have come up with something based on this blog post.
Controller
#RestController
public class AppController {
#SuppressWarnings("unchecked")
#RequestMapping(value = "/test3", method = RequestMethod.GET)
public Flux<String> index3() {
ElasticAdapter es = new ElasticAdapter();
JSONObject json = new JSONObject();
json.put("TestDoc", "Stack123");
Flux<String> fluxResponse = es.bulkIndex(json);
return fluxResponse;
}
ElasticAdapter
#Component
class ElasticAdapter {
String indexName = "test2";
private final RestHighLevelClient client;
private final ObjectMapper mapper;
private int processed = 1;
Flux<String> bulkIndex(JSONObject doc) {
return bulkIndexDoc(doc)
.doOnError(e -> System.out.print("Unable to index {}" + doc+ e));
}
private Flux<String> bulkIndexDoc(JSONObject doc) {
return Flux.create(sink -> {
try {
doBulkIndex(doc, bulkListenerToSink(sink));
} catch (JsonProcessingException e) {
sink.error(e);
}
});
}
private void doBulkIndex(JSONObject doc, BulkProcessor.Listener listener) throws JsonProcessingException {
System.out.println("Going to submit index request");
BiConsumer<BulkRequest, ActionListener<BulkResponse>> bulkConsumer =
(request, bulkListener) ->
client.bulkAsync(request, RequestOptions.DEFAULT, bulkListener);
BulkProcessor.Builder builder =
BulkProcessor.builder(bulkConsumer, listener);
builder.setBulkActions(10);
BulkProcessor bulkProcessor = builder.build();
// Submitting 5,000 index requests ( repeating same JSON)
for (int i = 0; i < 5000; i++) {
IndexRequest indexRequest = new IndexRequest(indexName, "person", i+1+"");
String json = doc.toJSONString();
indexRequest.source(json, XContentType.JSON);
bulkProcessor.add(indexRequest);
}
System.out.println("Submitted all docs
}
private BulkProcessor.Listener bulkListenerToSink(FluxSink<String> sink) {
return new BulkProcessor.Listener() {
#Override
public void beforeBulk(long executionId, BulkRequest request) {
}
#SuppressWarnings("unchecked")
#Override
public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
for (BulkItemResponse bulkItemResponse : response) {
JSONObject json = new JSONObject();
json.put("id", bulkItemResponse.getResponse().getId());
json.put("status", bulkItemResponse.getResponse().getResult
sink.next(json.toJSONString());
processed++;
}
if(processed >= 5000) {
sink.complete();
}
}
#Override
public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
failure.printStackTrace();
sink.error(failure);
}
};
}
public ElasticAdapter() {
// Logic to initialize Elasticsearch Rest Client
}
}
I used FluxSink to create the Flux of Responses to send back to the Client. At this point, I have no idea whether this correct or not.
My expectation is that the calling client should receive the responses in batches of 10 ( because bulk processor processess it in batches of 10 - builder.setBulkActions(10); ). I tried to consume the endpoint using Spring Webflix Client. But unable to work it out. This is what I tried
WebClient
public class FluxClient {
public static void main(String[] args) {
WebClient client = WebClient.create("http://localhost:8080");
Flux<String> responseFlux = client.get()
.uri("/test3")
.retrieve()
.bodyToFlux(String.class);
responseFlux.subscribe(System.out::println);
}
}
Nothing is printing on console as I expected. I tried to use System.out.println(responseFlux.blockFirst());. It prints all the responses as a single batch at the end and not in batches at .
If my approach is correct, what is the correct way to consume it? For the solution in my mind, this client will reside is another Webapp.
Notes: My understanding of Reactor API is limited. The version of elasticsearch used is 6.8.
So made the following changes to your code.
In ElasticAdapter,
public Flux<Object> bulkIndex(JSONObject doc) {
return bulkIndexDoc(doc)
.subscribeOn(Schedulers.elastic(), true)
.doOnError(e -> System.out.print("Unable to index {}" + doc+ e));
}
Invoked subscribeOn(Scheduler, requestOnSeparateThread) on the Flux, Got to know about it from, https://github.com/spring-projects/spring-framework/issues/21507
In FluxClient,
Flux<String> responseFlux = client.get()
.uri("/test3")
.headers(httpHeaders -> {
httpHeaders.set("Accept", "text/event-stream");
})
.retrieve()
.bodyToFlux(String.class);
responseFlux.delayElements(Duration.ofSeconds(1)).subscribe(System.out::println);
Added "Accept" header as "text/event-stream" and delayed Flux elements.
With the above changes, was able to get the response in real time from the server.
I am new to Java and Vert.x
I am Using vertx 3.5.3
I am trying to implement basic Authentication so that i can log into the remote sever before sending the post request
I tried the below code which is giving a compiler error .
I am getting a compiler error .
The method basicAuthentication(String, String) is undefined for the type HttpRequest What am i doing wrong .
The requirement is that i need to send user id and password for the remote server and access end point with a post request using a json Post body using Vert.x web client
I tried the below code
package com.aexp.csrt.qs.cb.resources;
import com.aexp.csrt.qs.models.cb.passTrou.SourceQueryModel;
import io.vertx.reactivex.ext.web.RoutingContext;
import io.vertx.reactivex.core.Vertx;
import io.vertx.ext.web.client.WebClient;
import io.vertx.ext.web.client.WebClientOptions;
import io.vertx.core.json.JsonObject;
import io.vertx.ext.web.client.HttpResponse;
import io.vertx.core.buffer.Buffer;
public class QueryExecutorFTS implements QueryExecutor {
private WebClient webClient;
#Override
public void executeQuery(SourceQueryModel Qm, RoutingContext rc) {
WebClientOptions options = new WebClientOptions().setMaxPoolSize(10).setConnectTimeout(5000)
.setVerifyHost(false);
JsonObject jreq = new JsonObject(Qm.getSourceQuery().getSourceDsl().getQuery());
Vertx vertx = rc.vertx();
webClient = WebClient.create(vertx.getDelegate(), options);
webClient
.basicAuthentication("myid", "mypassword")
.post(8094, "lpdospdb51079.phx.aexp.com", "/api/index/Text_search_name_idx/query")
.sendJsonObject(jreq,
ar -> {
if (ar.succeeded()) {
HttpResponse<Buffer> response = ar.result();
rc.response().setStatusCode(200).end(response.bodyAsString());
} else {
ar.cause().printStackTrace();
rc
.response()
.setStatusCode(500)
.setStatusMessage(ar.cause().getMessage())
.end();
}
})
;
}
}
Assuming you using the session handler on the server side like so:
router.route().handler(CookieHandler.create());
router.route().handler(SessionHandler.create(LocalSessionStore.create(vertx)));
router.route().handler(UserSessionHandler.create(authProvider));
you now need to first authenticate your client with the url you defined for login.
your login path -> /login
router.route(HttpMethod.POST, path)
.handler(FormLoginHandler.create(authProvider));
you normally POST the user / pass JSON
now on the client side you need two steps
POST your cred to the login URL
MultiMap form = MultiMap.caseInsensitiveMultiMap();
form.add("username", user);
form.add("password", password);
client
.post(port, "localhost", "/login")
.putHeader("content-type", "multipart/form-data")
.sendForm(form, ar -> {
if (ar.succeeded()) {
log.info("RESOURCES: {}", ar.result().bodyAsString());
log.info("COOKIES: {}", ar.result().cookies());//.getHeader("vertx-web.session"));
ar.result().cookies().stream().filter(c -> c.startsWith("vertx-web.session")).findFirst().ifPresent(s -> {
// do something with the session cookie - >cookie.accept(s.substring(0, s.indexOf(";")));
});
} else {
log.error("Something went wrong", ar.cause());
}
});
once you have the cookie you can call the server again.
client.
.get(port, "localhost", "/resource")
.putHeader("Cookie", cookie)
.send(ar -> { ...
I want to request to a url using okhttp in another thread (like IO thread) and get Response in the Android main thread, But I don't know how to create an Observable.
It's easier and safer to use Observable.defer() instead of Observable.create():
final OkHttpClient client = new OkHttpClient();
Observable.defer(new Func0<Observable<Response>>() {
#Override public Observable<Response> call() {
try {
Response response = client.newCall(new Request.Builder().url("your url").build()).execute();
return Observable.just(response);
} catch (IOException e) {
return Observable.error(e);
}
}
});
That way unsubscription and backpressure are handled for you. Here's a great post by Dan Lew about create() and defer().
If you wished to go the Observable.create() route then it should look more like in this library with isUnsubscribed() calls sprinkled everywhere. And I believe this still doesn't handle backpressure.
I realise this post is a bit old, but there's a new and more convenient way of doing this now
Observable.fromCallable {
client.newCall(Request.Builder().url("your url").build()).execute()
}
More info: https://artemzin.com/blog/rxjava-defer-execution-of-function-via-fromcallable/
I came late to the discussion but, if for some reason the code need to stream the response body, then defer or fromCallable won't do it. Instead one can employ the using operator.
Single.using(() -> okHttpClient.newCall(okRequest).execute(), // 1
response -> { // 2
...
return Single.just((Consumer<OutputStream>) fileOutput -> {
try (InputStream upstreamResponseStream = response.body().byteStream();
OutputStream fileOutput = responseBodyOutput) {
ByteStreams.copy(upstreamResponseStream, output);
}
});
},
Response::close, // 3
false) // 4
.subscribeOn(Schedulers.io()) // 5
.subscribe(copier -> copier.accept(...), // 6
throwable -> ...); // 7
The first lambda executes the response after upon subscription.
The second lambda creates the observable type, here with Single.just(...)
The third lambda disposes the response. With defer one could have used the try-with-resources style.
Set the eager toggle to false to make the disposer called after the terminal event, i.e. after the subscription consumer has been executed.
Of course make the thing happen on another threadpool
Here's the lambda that will consume the response body. Without eager set to false, the code will raise an IOException with reason 'closed' because the response will be already closed before entering this lambda.
The onError lambda should handle exceptions, especially the IOException that cannot be anymore caught with the using operator as it was possible with a try/catch with defer.
Okhttp3 with RxSingle background API call.
Disposable disposables = Single.fromCallable(() -> {
Log.e(TAG, "clearData: Thread[" + Thread.currentThread().getName() + "]");
OkHttpClient client = Util.getHttpClient();
Request request = new Request.Builder()
.addHeader("Authorization", "Bearer " + Util.getUserToken())
.url(BuildConfig.BASE_URL + ApiConstants.DELETE_FEEDS)
.build();
Response response = client.newCall(request).execute();
if(response.isSuccessful()) {
...
return ; // Any type
} else {
return ; // Any type
}
}).subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe((result) -> {
Log.d(TAG, "api() completed");
});
compositeDisposable.add(disposables);