Please bear with me, i dont usually use spring and havent used newer versions of java (I say newer I mean anything past prob 1.4)
Anyway, I have an issue where I have to do rest calls to do a search using multiple parallel requests. Ive been looking around online and I see you can use CompletableFuture.
So I created my method to get the objects I need form the rest call like:
#Async
public CompletableFuture<QueryObject[]> queryObjects(String url)
{
QueryObject[] objects= restTemplate.getForObject(url, QueryObject[].class);
return CompletableFuture.completedFuture(objects);
}
Now I need to call that with something like:
CompletableFuture<QueryObject> page1 = QueryController.queryObjects("http://myrest.com/ids=[abc, def, ghi]);
CompletableFuture<QueryObject> page2 = QueryController.queryObjects("http://myrest.com/ids=[jkl, mno, pqr]);
The problem I have is that the call needs to only do three ids at a time and there could be a list of variable number ids. So I parse the idlist and create a query string like above. The problem with that I am having is that while I can call the queries I dont have separate objects that I can then call CompletableFuture.allOf on.
Can anyone tell me the way to do this? Ive been at it for a while now and Im not getting any further than where I am now.
Happy to provide more info if the above isnt sufficient
You are not getting any benefit of using the CompletableFuture in a way you're using it right now.
The restTemplate method you're using is a synchronous method, so it has to finish and return a result, before proceeding. Because of that wrapping the final result in a CompletableFuture doesn't cause it to be executed asynchronously (neither in parallel). You just wrap a response, that you have already retrieved.
If you want to benefit from the async execution, then you can use for example the AsyncRestTemplate or the WebClient .
A simplified code example:
public ListenableFuture<ResponseEntity<QueryObject[]>> queryForObjects(String url) {
return asyncRestTemplate.getForEntity(url, QueryObject[].class);
}
public List<QueryObject> queryForList(String[] elements) {
return Arrays.stream(elements)
.map(element -> queryForObjects("http://myrest.com/ids=[" + element + "]"))
.map(future -> future.completable().join())
.filter(Objects::nonNull)
.map(HttpEntity::getBody)
.flatMap(Arrays::stream)
.collect(Collectors.toList());
}
Related
I am currently working on a project where I need to create a loop using spring webflux to generate a Flux for downstream processing. The loop should sequentially take batches of elements from a source (in this instance a repository) and pass the elements as signal in a Flux. To acquire the elements, we have a repository method which fetches the next batch. When all elements have been processed, the method yields an empty List.
I have identified that I can use Flux::generate in the following manner:
Flux.<List<Object>>generate(sink -> {
List<Object> batch = repository.fetch();
if (batch.isEmpty()) {
sink.complete();
} else {
sink.next(batch);
}
})
...
However, when I use this, the argument method runs continuously, buffering until I run out of memory.
I have also tried using Flux::create, but I am struggling to find an appropriate approach. I have found that I can do the following:
Consumer<Integer> sinker;
Flux<?> createFlux() {
return Flux.<List<Object>>create(sink -> sinker = integer -> {
List<Object> batch = repository.fetch();
if (batch.isEmpty()) {
sink.complete();
} else {
sink.next(batch);
}
})
...
.doOnNext(x -> sinker.accept(y))
...
}
Then I just need to call the Consumer initially to initiate the loop.
However, I feel like I am overly complicating a job which should have a fairly standard implementation. Also, this implementation requires secondary calls to get started, and I haven't found a decent way to initiate it within the pipeline (for instance, using .onSubscribe() doesn't work, as it attempts to call the Consumer before it has been assigned).
So in summary, I am looking for a simple way to create an unbounded loop while controlling the backpressure to avoid outOfMemory-errors.
I believe I have found a simpler solution which serves my need. The method Mono::repeat(BooleanSuplier) allows me to loop until the list is empty, simply by:
Mono.fromCallable(() -> repository.nextBatch())
.flatMap(/* do some stuff here */)
.repeat(() -> repository.hasNext())
If other more elegant solutions exist, I am still open for suggestions.
Im trying to do something simple with JavaRx, even tho the methods exist, they dont work as I thought and Im a little stuck with it.
Basically, I have something equivalent to:
Observable<String> filteredNames = Single.just("Jack")
.toObservable()
.filter(it -> "John".equals(it));
A call to an API (that returns a Single type and later I make it an observable so I can filter it, based on that filter I could return a value or not.
The expected output its to have a Mayble
So, Ive done this:
filteredNames.singleElement();
But seems like there is an error, neither firstElement();
No converter found for return value of type: class io.reactivex.internal.operators.observable.ObservableSingleMaybe.
I tried also
Maybe.just(filteredNames);
But its not possible since filteredNames has an Observable and not a String... Any idea how to do this?
I use JavaRx 2
You should not need to transform the Single to an Observable in order to filter it.
Single already supports the filter operator:
Maybe<String> m = Single.just("Jack")
.filter(it -> "John".equals(it));
The error is a hint that you are returning the Observable as a result being serialized / or converted which I would assume being a API endpoint.
If this shows to be the case, make sure to return the framework supported type, e.g. for a Spring Webflux endpoint, you would return a Mono:
return Mono.from(Single.just("Jack").filter(it -> "John".equals(it)).toFlowable());
My Problem:
I am working on an application using Spring Cloud data flow and Java. I am currently using a function called executionListByTaskName(String taskName) inside of my DataFlowTemplate's task operations. Then I use .getContent() to retrieve the collection of executions belonging to the task specified with "taskName". So I'm returning a Collection<TaskExecutionResource> from DataFlowTemplate.taskOperations().executionListByTaskName(the-task-name).getContent(). The problem with this function is that it will always retrieve the most recent 20 task executions. What I really want to implement is a function that when given two additional parameters, page number and task executions per page, will return the appropriate "page" of task executions. For example, if the page number was "2" and the size per page was "10," then I would return a collection of the 20th-30th most recent task executions.
My Solution (that isn't efficient hence why I'm on stack overflow):
I was able to get a "page" by using the collection of TaskExecutionResources and the Page, Pageable, and PageImpl classes given by the Spring framework. I converted the collection into an ArrayList by doing List<TaskExecutionResource> taskExecList = new ArrayList<>(taskExecColl). Then I created a Pageable object the standard way, taking into account of the provided page number and task executions per page. I create a Page<TaskExecutionResource> by creating a new PageImpl instance with the Pageable I created and the ArrayList of task execution resources. This works. However, converting the collection into an ArrayList is not scalable. The problem is, the underlying data structure of the collection is not a List of any sort, so i can't just cast it. If this is incorrect please let me know. Therefore, I don't know of any other way of implementing this pagination besides the solution above.
My Question:
Going through the DataFlowTemplate class, I found out that dataFlowTemplate.taskOperations().executionListByTaskName(taskName) calls a function in the TaskTemplate class.
Here is that function:
public org.springframework.cloud.dataflow.rest.resource.TaskExecutionResource.Page executionListByTaskName(String taskName) {
return (org.springframework.cloud.dataflow.rest.resource.TaskExecutionResource.Page)this.restTemplate.getForObject(this.executionByNameLink.expand(new Object[]{taskName}).getHref(), org.springframework.cloud.dataflow.rest.resource.TaskExecutionResource.Page.class, new Object[0]);
}
which in turn calls a function in the RestTemplate:
public <T> T getForObject(String url, Class<T> responseType, Object... uriVariables) throws RestClientException {
RequestCallback requestCallback = this.acceptHeaderRequestCallback(responseType);
HttpMessageConverterExtractor<T> responseExtractor = new HttpMessageConverterExtractor(responseType, this.getMessageConverters(), this.logger);
return this.execute(url, HttpMethod.GET, requestCallback, responseExtractor, (Object[])uriVariables);
}
What I want to know is what is the significance of the second parameter in these functions, Class<T> responseType which is set to org.springframework.cloud.dataflow.rest.resource.TaskExecutionResource.Page.class by default. I can only imagine that TaskExecutionResource.Page.class is the reason why only the most recent 20 TaskExecutionResources are returned but I am not sure. Also, is there any way to replace this second parameter so that I can return specific pages of TaskExecutionResources. I have tried different things regarding this but couldn't get anything working. If anybody has some insight on this, or knows of another solution to this problem, please let me know. Any help is appreciated.
I'm new with webflux and I'm trying to execute multiple monos with Flux. But i think i'm doing it wrong.. is this the best approach to execute multiple Mono and collect it to list?
Here is my code:
mainService.getAllBranch()
.flatMapMany(branchesList -> {
List<Branch> branchesList2 = (List<Branch>) branchesList.getData();
List<Mono<Transaction>> trxMonoList= new ArrayList<>();
branchesList2.stream().forEach(branch -> {
trxMonoList.add(mainService.getAllTrxByBranchId(branch.branchId));
});
return Flux.concat(trxMonoList); // <--- is there any other way than using concat?
})
.collectList()
.flatMap(resultList -> combineAllList());
interface MainService{
Mono<RespBody> getAllBranch();
Mono<RespBody> getAllTrxByBranchId(String branchId); //will return executed url ex: http://trx.com/{branchId}
}
so far my with above code i can explain it like this:
Get all branches
iterate through all branchesList2 and add it to trxMonoList
return Flux.concat, this is where i'm not sure is this the right way or not. but it's working
combine all list
I'm just confuse is this the proper way to use Flux in my context? or is there any better way to achieve with what i'm trying to do?
You need to refactor a little bit your code to reactive.
mainService.getAllBranch()
.flatMapMany(branchesList -> Flux.fromIterable(branchesList.getData())) (1)
.flatMap(branch -> mainService.getAllTrxByBranchId(branch.branchId)) (2)
.collectList()
.flatMap(resultList -> combineAllList());
1) Create Flux of branches from List;
2) Iterate through the each element and call a service.
You shouldn't use Stream API in Reactor because it has the same methods but with adaptaptions and optimisations for multythreading.
The real problem here is that you shouldn't be hitting a Mono multiple times within a Flux. That will give you problems. If you are designing the API you should fix that to do what you want in a correct reactive manner.
interface MainService{
Flux<Branch> getAllBranch();
Flux<Transaction> getAllTrxByBranchId(Flux<String> branchIds);
}
Then your code becomes simpler and the reactive framework will work properly.
mainService.getAllTrxByBranchId(mainService.getAllBranch().map(Branch::getId));
While working with Spring Webflux, I'm trying to insert some data in the realm object server which interacts with Java apps via a Rest API. So basically I have a set of students, who have a set of subjects and my objective is to persist those subjects in a non-blocking manner. So I use a microservice exposed via a rest endpoint which provides me with a Flux of student roll numbers, and for that flux, I use another microservice exposed via a rest endpoint that gets me the Flux of subjects, and for each of these subjects, I want to persist them in the realm server via another rest endpoint. I wanted to make this all very nonblocking which is why I wanted my code to look like this.
void foo() {
studentService.getAllRollnumbers().flatMap(rollnumber -> {
return subjectDirectory.getAllSubjects().map(subject -> {
return dbService.addSubject(subject);
})
});
}
But this doesn't work for some reason. But once I call blocks on the things, they get into place, something like this.
Flux<Done> foo() {
List<Integer> rollNumbers = studentService.getAllRollnumbers().collectList().block();
rollNumbers.forEach(rollNumber -> {
List<Subject> subjects = subjectDirectory.getAllSubjects().collectList().block();
subjects.forEach(subject -> {dbService.addSubject(subject).block();});
});
return Flux.just(new NotUsed());
}
getAllRollnumbers() returns a flux of integers.
getAllSubjects() returns a flux of subject.
and addSubject() returns a Mono of DBResponse pojo.
What I can understand is that the thread executing this function is getting expired before much of it gets triggerred. Please help me work this code in an async non blocking manner.
You are not subscribing to the Publisher at all in the first instance that is why it is not executing. You can do this:
studentService.getAllRollnumbers().flatMap(rollnumber -> {
return subjectDirectory.getAllSubjects().map(subject -> {
return dbService.addSubject(subject);
})
}).subscribe();
However it is usually better to let the framework take care of the subscription, but without seeing the rest of the code I can't advise.