Is there an operator in RxJava, an external library or a way I'm missing to create a flowable/observable that recieves a function that controls the emission of data, like a valve?
I have a huge json file I need to process but I have to get a portion of the file, a list of entities, process it and then get another portion, I have tried using windows(), buffer() but the BiFunction I pass to Flowable.generate() keeps executing after I recieved the first list and I haven't finished processing it. I also tried FlowableTransformers.valve() from hu.akarnokd.rxjava3.operators but it just piles up the items before the flatMap() function that process the list
private Flowable<T> flowable(InputStream inputStream) {
return Flowable.generate(() -> jsonFactory.createParser(new GZIPInputStream(inputStream)), (jsonParser, emitter) -> {
final var token = jsonParser.nextToken();
if (token == null) {
emitter.onComplete();
}
if (JsonToken.START_ARRAY.equals(token) || JsonToken.END_ARRAY.equals(token)) {
return jsonParser;
}
if (JsonToken.START_OBJECT.equals(token)) {
emitter.onNext(reader.readValue(jsonParser));
}
return jsonParser;
}, JsonParser::close);
}
Edit: I need to control de emission of items to don't overload the memory and the function that process the data, because that function reads and writes to database, also the processing needs to be sequentially. The function that process the data it's not entirely mine and it's written in RxJava and it's expected that I use Rx.
I managed to solve it like this but if there is another way let me know please:
public static <T> Flowable<T> flowable(InputStream inputStream, JsonFactory jsonFactory, ObjectReader reader, Supplier<Boolean> booleanSupplier) {
return Flowable.generate(() -> jsonFactory.createParser(new GZIPInputStream(inputStream)), (jsonParser, emitter) -> {
if (booleanSupplier.get()) {
final var token = jsonParser.nextToken();
if (token == null) {
emitter.onComplete();
}
if (JsonToken.START_ARRAY.equals(token) || JsonToken.END_ARRAY.equals(token)) {
return jsonParser;
}
if (JsonToken.START_OBJECT.equals(token)) {
emitter.onNext(reader.readValue(jsonParser));
}
}
return jsonParser;
}, JsonParser::close);
}
Edit2: This is one of the ways I'm currently consuming the function
public Flowable<List<T>> paging(Function<List<T>, Single<List<T>>> function) {
final var atomicInteger = new AtomicInteger(0);
final var atomicBoolean = new AtomicBoolean(true);
return flowable(inputStream, jsonFactory, reader, atomicBoolean::get)
.buffer(pageSize)
.flatMapSingle(list -> {
final var counter = atomicInteger.addAndGet(1);
if (counter == numberOfPages) {
atomicBoolean.set(false);
}
return function.apply(list)
.doFinally(() -> {
if (atomicInteger.get() == numberOfPages) {
atomicInteger.set(0);
atomicBoolean.set(true);
}
});
});
}
Managed to solve it like this
public static Flowable<Object> flowable(JsonParser jsonParser, ObjectReader reader, PublishProcessor<Boolean> valve) {
return Flowable.defer(() -> {
final var token = jsonParser.nextToken();
if (token == null) {
return Completable.fromAction(jsonParser::close)
.doOnError(Throwable::printStackTrace)
.onErrorComplete()
.andThen(Flowable.empty());
}
if (JsonToken.START_OBJECT.equals(token)) {
final var value = reader.readValue(jsonParser);
final var just = Flowable.just(value).compose(FlowableTransformers.valve(valve, true));
return Flowable.concat(just, flowable(jsonParser, reader, valve));
}
return flowable(jsonParser, reader, valve);
});
}
Related
I am using Vert.x in my project, I used future() to get the results from a MongoDB query. However when I do future().result it returns "null". I want the result to be saved in the future and I will use it for other APIs. Is there any guide for me, I will be very grateful and appreciate if someone give me a solution. Thanks
router.class
rivate void getClazzById(RoutingContext rc) {
Future<JsonObject> future = Future.future();
String clazzId = rc.request().getParam("clazzId");
classService.getClazzById(clazzId, res -> {
System.out.println(res.result());
if (res.succeeded()) {
JsonObject result = res.result();
if (result != null) {
future.complete(res.result());
rc.response().setStatusCode(200).putHeader("Content-Type", "application/json; charset=utf-8")
.end(result.encodePrettily());
} else {
rc.response().setStatusCode(400).putHeader("Content-Type", "application/json; charset=utf-8")
.end(new JsonObject().put("error", "Class not found!").encodePrettily());
}
} else
rc.fail(res.cause());
});
future.setHandler(s -> {
if (s.succeeded()) {
System.out.println("sss: " +s.result()); // print: {"_id":"123", "name":"abc"}
}
else {
System.out.println("fail");
}
});
System.out.println("hhhhhh: " + future.result()); // I want to print {"_id":"123", "name":"abc"}
}
service.class
public void getClazzById(String clazzId, Handler<AsyncResult<JsonObject>> resultHandler) {
JsonObject query = new JsonObject().put("_id", clazzId);
client.findOne(Collection.CLAZZ, query, null, ar -> {
if (ar.succeeded()) {
if (ar.succeeded()) {
JsonObject result = ar.result();
if (result == null) {
resultHandler.handle(Future.failedFuture("Error"));
} else {
resultHandler.handle(Future.succeededFuture(result));
}
} else {
ar.cause();
}
}
});
}
When writing asynchronous code, you are carried to use the framework / runtime semantics and tools for communication.
You are already leveraging one of Vert.x's way of async communication, the Future - but in the wrong manner trying to retrieve its result inline.
Instead of having the Future result accessed within your method, you need to return it as your mean of communication to a caller, which would be able to set a completion handler (Handler) to it:
private Future<JsonObject> getClazzById(RoutingContext rc) {
Future<JsonObject> future = Future.future();
String clazzId = rc.request().getParam("clazzId");
classService.getClazzById(clazzId, res -> {
if (res.succeeded()) {
JsonObject result = res.result();
if (result != null) {
future.complete(res.result()); // set the retrieved result
rc.response().setStatusCode(200).putHeader("Content-Type", "application/json; charset=utf-8")
.end(result.encodePrettily());
} else {
future.complete(null); // you need to provide 'null' result to avoid caller blocking
rc.response().setStatusCode(400).putHeader("Content-Type", "application/json; charset=utf-8")
.end(new JsonObject().put("error", "Class not found!").encodePrettily());
}
} else
rc.fail(res.cause());
});
return future; // return the future to caller
}
An interested caller would be able to set a handler for Future completion as needed:
getClassById(rc).setHandler(s -> {
if (s.succeeded()) {
System.out.println("sss: " +s.result()); // print: {"_id":"123", "name":"abc"}
}
else {
System.out.println("fail");
}
});
As a side note: you are not setting the API boundaries properly in your business logic as you are trying to resolve the HTTP Response result which generally denotes the request processing end while still returning the query result to be handled in some other manner.
I have written a logic using spring reactor library to get all operators and then all devices for each operator (paginated) in async mode.
Created a flux to get all operator and then subscribing to it.
final Flux<List<OperatorDetails>> operatorDetailsFlux = reactiveResourceProvider.getOperators();
operatorDetailsFlux
.subscribe(operatorDetailsList -> {
for (final OperatorDetails operatorDetails : operatorDetailsList) {
getAndCacheDevicesForOperator(operatorDetails.getId());
}
});
Now, for each operator I'm fetching the devices which requires multiple subscriptions to get device mono which gets all pages async by subscribing to the MONO.
private void getAndCacheDevicesForOperator(final int operatorId) {
Mono<DeviceListResponseEntity> deviceListResponseEntityMono = reactiveResourceProvider.getConnectedDeviceMonoWithRetryAndErrorSpec(
operatorId, 0);
deviceListResponseEntityMono.subscribe(deviceListResponseEntity -> {
final PaginatedResponseEntity PaginatedResponseEntity = deviceListResponseEntity.getData();
final long totalDevicesInOperator = PaginatedResponseEntity.getTotalCount();
int deviceCount = PaginatedResponseEntity.getCount();
while (deviceCount < totalDevicesInOperator) {
final Mono<DeviceListResponseEntity> deviceListResponseEntityPageMono = reactiveResourceProvider.getConnectedDeviceMonoWithRetryAndErrorSpec(
operatorId, deviceCount);
deviceListResponseEntityPageMono.subscribe(deviceListResponseEntityPage -> {
final List<DeviceDetails> deviceDetailsList = deviceListResponseEntityPage.getData()
.getItems();
// work on devices
});
deviceCount += DEVICE_PAGE_SIZE;
}
});
}
This code works fine. But my question is it a good idea to subscribe to mono from inside subscribe?
I broke it down to two flows 1st getting all operators and then getting all devices for each operator.
For pagination I'm using Flux.expand to extract all pages.
public Flux<OperatorDetails> getAllOperators() {
return getOperatorsMonoWithRetryAndErrorSpec(0)
.expand(paginatedResponse -> {
final PaginatedEntity operatorDetailsPage = paginatedResponse.getData();
if (morePagesAvailable(operatorDetailsPage) {
return getOperatorsMonoWithRetryAndErrorSpec(operatorDetailsPage.getOffset() + operatorDetailsPage.getCount());
}
return Mono.empty();
})
.flatMap(responseEntity -> fromIterable(responseEntity.getData().getItems()))
.subscribeOn(apiScheduler);
}
public Flux<Device> getAllDevices(final int opId, final int offset) {
return getConnectedDeviceMonoWithRetryAndErrorSpec(opId, offset)
.expand(paginatedResponse -> {
final PaginatedEntity deviceDetailsPage = paginatedResponse.getData();
if (morePagesAvailabile(deviceDetailsPage)) {
return getConnectedDeviceMonoWithRetryAndErrorSpec(opId,
deviceDetailsPage.getOffset() + deviceDetailsPage.getCount());
}
return Mono.empty();
})
.flatMap(responseEntity -> fromIterable(responseEntity.getData().getItems()))
.subscribeOn(apiScheduler);
}
Finally I'm creating a pipeline and subscribing to it to trigger the pipeline.
operatorDetailsFlux
.flatMap(operatorDetails -> {
return reactiveResourceProvider.getAllDevices(operatorDetails.getId(), 0);
})
.subscribe(deviceDetails -> {
// act on devices
});
I want to create a unit test for a client service.
The function of the client service is to call the webservice, get data, and update the database as scheduled.
The scheduled method return void.
How to create unit test for a
client service
schedule method
void returning methods
The client is like this:
#ApplicationScoped
public class ClientClass {
private static final Logger LOGGER = Logger.getLogger(VdsClient.class);
#Inject
Client client;
VMR vmr;
CommandService commandService;
public VdsClient(VMR vmr,
CommandService commandService) {
this.vmr = vmr;
this.commandService = commandService;
}
#Scheduled(XXX)
public void getVal() {
var monitoringStateFilter =
new VMF.vmf(true, true);
var monoResultList =
vmr.fvms(monitoringStateFilter)
.collectList();
if (monoResultList != null) {
var resultList = monoResultList.block();
if (resultList != null) {
resultList.stream()
.map(row -> row.getValue("val", val.class))
.map(vin -> this.updateEstimate(val.getValue()))
}
}
}
public Tuple2<String, Boolean> updateEstimate(String val) {
List<route> routeList;
try {
routeList = vdsClient.getval(val)
.getItem();
boolean hasDealerDestination = false;
for (Route route : routeList) {
if (vd.DestLocationType._00.value()
.equals(route.getTransportConnectionPointTyp())) {
hasDealerDestination = true;
var estimate = DateTimeUtil.convertToInstantWithOffset(route.getArrivalDate(),
route.getArrivalTime(), route.getTimezone(), DateTimeUtil.PATTERN_LONG_DATE);
if (estimate == null) {
return Tuples.of(val, false);
}
var result = this.updateVehicleEstimate(val, estimate);
return Tuples.of(val, result);
}
}
if (!hasDealerDestination) {
return Tuples.of(val, false);
} else {
return Tuples.of(val, false);
}
} catch (route e) {
return Tuples.of(val, false);
} catch (Exception e) {
return Tuples.of(val, false);
}
}
public Boolean updateVehicleEstimate(String val, Instant estimate) {
var vehicleUpdate = vu.vuc.builder()
.val(new Val(val))
.Estimate(estimate)
.build();
return (Boolean) cs.ec(vu).block();
}
A unit should only be testing that a particular unit of code is working fine. In your case, for the unit test, you should assume that the webservice will return the data and the db updation works fine. We can accomplish this through mocking the response for each of these calls.
For void returning methods, you can actually verify if the call was indeed made or not.
For example:
Mockito.verify(mockedObject, Mockito.times(1)).callWebService(mockedParameter1, mockedParameter2);
There is another way, though I personally don't prefer that:
You can declare a class variable and make sure the value updates itself whenever the scheduled method reaches the end of the code. Read that value in your test and assert on its value. If the value is updated, then your code worked fine, else NO and it's a failure.
Also, in case you want to actually make sure the webservice returned the correct response / db entry was updated, then those should be part of integration tests and not unit tests.
I have the following rest controller, which receives requests, transforms them into JSON strings and puts them into a concurrent queue.
I would like to make a Flux out of this queue and subscribe to it.
Unfortunately, it doesn't work.
What am I doing wrong here?
#RestController
public class EventController {
private final ObjectMapper mapper = new ObjectMapper();
private final FirehosePutService firehosePutService;
private ConcurrentLinkedQueue<String> events = new ConcurrentLinkedQueue<>();
private int batchSize = 10;
#Autowired
public EventController(FirehosePutService firehosePutService) {
this.firehosePutService = firehosePutService;
Flux<String> eventFlux = Flux.create((FluxSink<String> sink) -> {
String next;
while (( next = events.poll()) != null) {
sink.next(next);
}
});
eventFlux.publish().autoConnect().subscribe(new BaseSubscriber<String>() {
int consumed;
List<String> batchOfEvents = new ArrayList<>(batchSize);
#Override
protected void hookOnSubscribe(Subscription subscription) {
request(batchSize);
}
#Override
protected void hookOnNext(String value) {
batchOfEvents.add(value);
consumed++;
if (consumed == batchSize) {
batchOfEvents.addAll(events);
log.info("Consume {} elements. Size of batchOfEvents={}", consumed, batchOfEvents.size());
firehosePutService.saveBulk(batchOfEvents);
consumed = 0;
batchOfEvents.clear();
events.clear();
request(batchSize);
}
}
});
}
#GetMapping(value = "/saveMany", produces = "text/html")
public ResponseEntity<Void> saveMany(#RequestParam MultiValueMap<String, String> allRequestParams) throws JsonProcessingException {
Map<String, String> paramValues = allRequestParams.toSingleValueMap();
String reignnEvent = mapper.writeValueAsString(paramValues);
events.add(reignnEvent);
return new ResponseEntity<>(HttpStatus.OK);
}
}
First of all, you use poll method. It is not blocking and returns null if queue is empty. You loop collection until first null (i.e. while (next != null), so your code exits loop almost immediately because queue is empty on start. You must replace poll with take which is blocking and will wait until element is available.
Secondly, hookOnNext is invoked when the event is removed from the queue. However, you are trying to read events again using batchOfEvents.addAll(events);. Moreover, you also clear all pending events events.clear();
I advise you to remove all direct access to events collection from hookOnNext method.
Why do you use Flux here at all? Seems overcomplicated. You can use plain thread here
#Autowired
public EventController(FirehosePutService firehosePutService) {
this.firehosePutService = firehosePutService;
Thread persister = new Thread(() -> {
List<String> batchOfEvents = new ArrayList<>(batchSize);
String next;
while (( next = events.take()) != null) {
batchOfEvents.add(value);
if (batchOfEvents.size() == batchSize) {
log.info("Consume {} elements. Size of batchOfEvents={}", consumed, batchOfEvents.size());
firehosePutService.saveBulk(batchOfEvents);
batchOfEvents.clear();
}
}
});
persister.start();
}
I am trying to return a CompletableFuture that will return a response from Amazon. My code first checks to see if a response is cached locally. If so, it returns the response, otherwise it calls Amazon. [Note: The real version will also cache the response received from Amazon, but I haven't included that as the code is already pretty complicated.]
Is there a way to change the implementation of my callAmazon method (or otherwise reorganize the code) so that I don't have to "manually" copy the response state from amazonApi to the finalResponse?
I don't want to return cacheCheck directly because I don't want the caller to be able to complete() it.
public CompletableFuture<Response> fetchProductList() {
CompletableFuture<Response> finalResponse = new CompletableFuture<>();
CompletableFuture<Response> cacheCheck = //...
// First, see if we have a cached copy
cacheCheck.whenComplete((response, throwable) -> {
if (throwable == null) {
// Cache hit. Return the cached response
finalResponse.complete(response);
} else {
// Cache miss. Call Amazon
callAmazon(finalResponse);
}
});
return finalResponse;
}
private void callAmazon(CompletableFuture<Response> finalResponse) {
CompletableFuture<Response> amazonApi = //...
amazonApi.whenComplete((response, throwable) -> {
// Copy the state to the `finalResponse`
if (throwable == null) {
finalResponse.complete(response);
} else {
finalResponse.completeExceptionally(throwable);
}
});
}
What makes your requirement so complex is the fact that cacheCheck can throw an exception.
What I would do in your case is to refactor the cache to deliver either null if the value was not found in the cache, or the actual Response, if the value was in the cache.
Furthermore, I would modify callAmazon to return directly the CompletableFuture:
private CompletableFuture<Response> callAmazon() {
CompletableFuture<Response> amazonApi = //...
return amazonApi;
}
This way you can use thenCompose:
final CompletableFuture<Response> cacheCheck = //...
final CompletableFuture<Response> amazonResponse = callAmazon();
final CompletableFuture<Response> finalResult =
cachedResponse.thenCompose(cacheResult -> {
return cacheResult == null ? amazonResponse : CompletableFuture.completedFuture(cacheResult);
});
If you really need to throw an exception from the cache, you can use exceptionally to convert the exception to a null value, and then use thenCompose to decide if you use the cache value, or call Amazon:
final CompletableFuture<Response> finalResult = cachedResponse.exceptionally(e -> {
return null;
}).thenCompose(cacheResult -> {
return cacheResult == null ? amazonResponse : CompletableFuture.completedFuture(cacheResult);
});