Develop Reference

Optimizing method with list of 500k+ elements

I'm looking for some help since I don't know how to optimize a process.
I have to invoke a service that returns a list with more than 500K elements (I don't know why, these services belongs to the client), per each element of the list, I have to invoke 2 more services and then save some attributes in our database, this last step is not the problem, but the entire process took between 1 and 2 seconds per element, so with this time is going to take like more of 100 hours to complete the process.
My approach is the following, I have my main method, inside this method I get the large list, then I use a parallelStream to iterate in the elements of the list and then I use a CompletableFuture to call the method that invokes the 2 services mentioned above. I've tried changing the parallelStream to stream and for-each , tried to split the main list into smaller lists and many other things but I don't see a better performance, I think the problem is the invocation of those 2 services but I want to try luck asking here.
I'm using java 11, spring, and for the invocation of the services I'm using RestTemplate, and this is my code:
public void updateDiscount() {
//List with 500k elements
var relationshipList = relationshipService.getLargeList();
//CompletableFuture to make the async calls to the method above
relationshipList.parallelStream().forEach(level1 -> {
CompletableFuture.runAsync(() -> relationshipService.asyncDiscountSave(level1));
});
}
//Second class
#Async("nameOfThePool")
public void asyncDiscountSave(ElementOfList element) {
//Logic to create request
//.........
var responseClients = anotherClass.getClients(element.getGroup1()) //get the first response with restTemplate
var responseProducts = anotherClass.getProducts(element.getGroup2())//get the second response with restTemplate
for (var client : responseClients) {
for (var product : responseProducts) {
//Here we just save some attributes of these objects on our DB
}
}
}
Thanks for the help.
UPDATE:
For this particular case, the only improvement that I can do is to pass a thread pool to the completable future, the problem is the response time of the services that I need to invoke.
I decided to follow a second approach and it took like 5 hours to complete, compared with the first approach this is acceptable.

As you haven't defined an executor you are using the default pool. Adding an executor allow you to create many threads as you needed and the server resources can manage
public void updateDiscount() {
Executor executor = Executors.newFixedThreadPool( 100 );//Define the number according to server resources performance
//List with 500k elements
var relationshipList = relationshipService.getLargeList();
//CompletableFuture to make the async calls to the method above
relationshipList.parallelStream().forEach(level1 -> {
CompletableFuture.runAsync(() -> relationshipService.asyncDiscountSave(level1), executor);
});
}

how to convert Flux<pojo> to ArrayList<String>

In my spring-boot springboot service class, I have created the following code which is not working as desired:
Service class:
Flux<Workspace> mWorkspace = webClient.get().uri(WORKSPACEID)
.retrieve().bodyToFlux(Workspace.class);
ArrayList<String> newmWorkspace = new ArrayList();
newmWorkspace = mWorkspace.blockLast();
return newmWorkspace;
Please someone help me on converting the list of json values to put it into arrayList
Json
[
{
"id:"123abc"
},
{
"id:"123abc"
}
]

Why is the code not working as desired
mWorkspace is a publisher of one or many items of type Workspace.
Calling newmWorkspace.blockLast() will get a Workspace from that Publisher:
which is an object of type: Workspace and not of type ArrayList<String>.
That's why : Type mismatch: cannot convert from Workspace to ArrayList<String>
Converting from Flux to an ArrayList
First of all, in reactive programming, a Flux is not meant to be blocked, the blockxxx methods are made for testing purposes. If you find yourself using them, then you may not need reactive logic.
In your service, you shall try this :
//initialize the list
ArrayList<String> newmWorkspace = new ArrayList<>();
Flux<Workspace> mWorkspace = webClient.get().uri(WORKSPACEID)
.retrieve().bodyToFlux(Workspace.class)
.map(workspace -> {
//feed the list
newmWorkspace.add(workspace.getId());
return workspace;
});
//this line will trigger the publication of items, hence feeding the list
mWorkspace.subscribe();
Just in case you want to convert a JSON String to a POJO:
String responseAsjsonString = "[{\"id\": \"123abc\"},{\"id\": \"123cba\"}] ";
Workspace[] workspaces = new ObjectMapper().readValue(responseAsjsonString, Workspace[].class);

You would usually want to avoid blocking in a non-blocking application. However, if you are just integrating from blocking to non-blocking and doing so step-by-step (unless you are not mixing blocking and non-blocking in your production code), or using a servlet stack app but want to only use the WebFlux client, it should be fine.
With that being said, a Flux is a Publisher that represents an asynchronous sequence of 1..n emitted items. When you do a blockLast you wait until the last signal completes, which resolves to a Workspace object.
You want to collect each resolved item to a list and return that. For this purpose, there is a useful method called collectList, which does this job without blocking the stream. You can then block the Mono<List<Workspace>> returned by this method to retrieve the list.
So this should give you the result you want:
List<Workspace> workspaceList = workspaceFlux.collectList().block();
If you must use a blocking call in the reactive stack, to avoid blocking the event loop, you should subscribe to it on a different scheduler. For the I/O purposes, you should use the boundedElastic Scheduler. You almost never want to call block on a reactive stack, instead subscribe to it. Or better let WebFlux to handle the subscription by returning the publisher from your controller (or Handler).

How to pass Mono<> result from previous step to the next doOnSuccess() method

Let's say that I have a method addVoteToSong like:
public Mono<Map<Song, VoteKind>> addVoteToSong(Principal principal, String songId, VoteKind voteKind) {
return
userRepository.findUserByUsername(principal.getName())
.doOnSuccess(song -> songRepository.findSongById(songId))
.doOnSuccess(vote -> voteRepository.add(Vote.builder().song()))
.//(the rest of the code)
}
I want to pass a result from the line:
userRepository.findUserByUsername(principal.getName())
and
.doOnSuccess(song -> songRepository.findSongById(songId))
to the built object in the line:
.doOnSuccess(vote -> voteRepository.add(Vote.builder().song(here result from findSongById).user(here result from findUserByUsername))
Here comes the question, is it possible to reuse previous API call result in the next doOnSuccess method or I should split find API calls at the same time, giving up on Reactor's cascading operations? On the internet, I have found examples with single save method without basing on the indirect result of the reactive stream and that's why question occurred. I will be grateful for suggestions on how to reach a goal.

Martin,
First of all, be aware that .doOnXXX are just callbacks that will be executed on some archived conditions. You should avoid putting a business logic inside of them.
Coming back to the question, the first idea that comes to my mind is to benefit from zip operator. So you have to put 2 publishers .findUserByUsername and .findSongById and combine the result using BiFunction. So you can try the following:
public Mono<Map<Song, VoteKind>> addVoteToSong(Principal principal, String songId, VoteKind voteKind) {
return Mono
.zip(
userRepository.findUserByUsername(principal.getName()),
songRepository.findSongById(songId),
(user, song) -> voteRepository.add(Vote.builder().song(song).user(user).build())
)
.flatMap(Function.identity())
// your code is here
}

JAVA 8 pass return value back into same method x number of times

I have this code which I want to refactor using a functional style, using Java 8. I would like to remove the mutable object currentRequest and still return the filtered request.
HttpRequest currentRequest = httpRequest;
for (Filter filter : filters) {
currentRequest = filter.doFilter(currentRequest);
}
The aim is to pass a request to the filter.doFilter method, and take the output and pass it back into the filter.doFilter method, and continue to do this until all filters are applied.
For example in a more convoluted way to the for loop
HttpRequest filteredRequest1 = filters.get(0).doFilter(currentRequest);
HttpRequest filteredRequest2 = filters.get(1).doFilter(filteredRequest1);
HttpRequest filteredRequest3 = filters.get(2).doFilter(filteredRequest2);
...
I think this is a case for composing functions, and the doFilter method should be a function like below:
Function<HttpRequest, HttpRequest> applyFilter = request -> filters.get(0).doFilter(request);
But I know this is totally wrong, as I got stuck here.
The other way I was thinking was to use reduce, but I cannot see a way of using it in this case.
If you could help me out with a way of doing this, or point me to some resource that will be great.

It looks like you may want to do a reduce with your HttpRequest as its identity. Each step of the reduce will combine the intermediate result with the next filter, like so:
filters.stream().reduce(currentRequest,
(req, filter) -> filter.doFilter(req),
(req1, req2) -> throwAnExceptionAsWeShouldntBeHere());
Note: the last function is used to merge two HttpRequests together if a parallel stream is used. If that's the route you wish to go down, then proceed with caution.

Here's a way that streams the filters and maps each one of them to a UnaryOperator<HttpRequest>. Then, all the functions are reduced via the Function.andThen operator and finally, if the filters collections wasn't empty, the resulting composed function is executed with the currentRequest as an argument:
HttpRequest result = filters.stream()
.map(filter -> ((Function<HttpRequest, HttpRequest>) filter::doFilter))
.reduce(Function::andThen)
.map(function -> function.apply(currentRequest))
.orElse(currentRequest);

How to pass in multiple arguments in Java Lambda BiFunctions using Async?

I'm working with a PlayFramework example but I'm having trouble with the asynchronous calls. The function thenCombineAsync seems to only take in a BiFunction. I'd like to pass in multiple arguments to my render function (such as vendors) but I'm not sure how to add more.
return entryRepository.lookup(id).thenCombineAsync(vendorsFuture(entryOptional, vendors) -> {
// This is the HTTP rendering thread context
Entry c = entryOptional.get();
Form<Entry> entryForm = formFactory.form(Entry.class).fill(c);
return ok(views.html.editForm.render(id, entryForm, vendors));
}, httpExecutionContext.current());