In my spring-boot springboot service class, I have created the following code which is not working as desired:
Service class:
Flux<Workspace> mWorkspace = webClient.get().uri(WORKSPACEID)
.retrieve().bodyToFlux(Workspace.class);
ArrayList<String> newmWorkspace = new ArrayList();
newmWorkspace = mWorkspace.blockLast();
return newmWorkspace;
Please someone help me on converting the list of json values to put it into arrayList
Json
[
{
"id:"123abc"
},
{
"id:"123abc"
}
]
Why is the code not working as desired
mWorkspace is a publisher of one or many items of type Workspace.
Calling newmWorkspace.blockLast() will get a Workspace from that Publisher:
which is an object of type: Workspace and not of type ArrayList<String>.
That's why : Type mismatch: cannot convert from Workspace to ArrayList<String>
Converting from Flux to an ArrayList
First of all, in reactive programming, a Flux is not meant to be blocked, the blockxxx methods are made for testing purposes. If you find yourself using them, then you may not need reactive logic.
In your service, you shall try this :
//initialize the list
ArrayList<String> newmWorkspace = new ArrayList<>();
Flux<Workspace> mWorkspace = webClient.get().uri(WORKSPACEID)
.retrieve().bodyToFlux(Workspace.class)
.map(workspace -> {
//feed the list
newmWorkspace.add(workspace.getId());
return workspace;
});
//this line will trigger the publication of items, hence feeding the list
mWorkspace.subscribe();
Just in case you want to convert a JSON String to a POJO:
String responseAsjsonString = "[{\"id\": \"123abc\"},{\"id\": \"123cba\"}] ";
Workspace[] workspaces = new ObjectMapper().readValue(responseAsjsonString, Workspace[].class);
You would usually want to avoid blocking in a non-blocking application. However, if you are just integrating from blocking to non-blocking and doing so step-by-step (unless you are not mixing blocking and non-blocking in your production code), or using a servlet stack app but want to only use the WebFlux client, it should be fine.
With that being said, a Flux is a Publisher that represents an asynchronous sequence of 1..n emitted items. When you do a blockLast you wait until the last signal completes, which resolves to a Workspace object.
You want to collect each resolved item to a list and return that. For this purpose, there is a useful method called collectList, which does this job without blocking the stream. You can then block the Mono<List<Workspace>> returned by this method to retrieve the list.
So this should give you the result you want:
List<Workspace> workspaceList = workspaceFlux.collectList().block();
If you must use a blocking call in the reactive stack, to avoid blocking the event loop, you should subscribe to it on a different scheduler. For the I/O purposes, you should use the boundedElastic Scheduler. You almost never want to call block on a reactive stack, instead subscribe to it. Or better let WebFlux to handle the subscription by returning the publisher from your controller (or Handler).
Related
Our project use a external library. It has a method return FluxMap (since FluxMap is not completely public so just call it Flux):
Flux<MappedType> aFluxMap = Library.createMappingToMappedType();
I have to emit some objects to aFluxMap to get them converted to MappedType (it has private constructor, few setter), then I can:
aFluxMap.doOnNext(converted -> doJob(converted))
I expect that there is a method on Flux/Mono like:
aFluxMap.emit(myObj);
But I could not find any method like that.
I have searched "how to emit to flux dynamically", then there is a solution:
FluxProcessor p = UnicastProcessor.create().serialize();
FluxSink sink = p.sink();
sink.next(mess);
But seem that it emit to newly created flux (p), not my aFluxMap. So I want to ask is there any way to emit message to a existed Flux (or how to connect a FluxSink to a existed Flux, so that whenever FluxSink .next(mess), then the existed Flux get the message). Thank you
Note: please don't pay much attention to the stupidity of the library. We must use it
==========================================
UPDATE:
As #lkatiforis suggestion:
FluxProcessor p = //see above
Flux<MappedType> aFluxMap = Library.createMappingToMappedType();
p.flatMap(raw -> aFluxMap).subscribe();
I got another issue. Library.createMappingToMappedType() return a subscribed Flux with its source is UnicastProcessor (also subscribed).
When I call p.flatMap(raw -> aFluxMap), then internally aFluxMap get subscribed again cause its source also get subscribed again, so that I got an exception telling that "UnicastProcessor can be subscribe once". Any suggestion?
You can create a new stream and then merge the two streams into one by using one of these methods: merge, concat, zip, and their variants.
Here is an example:
Flux<MappedType> yourFlux = //...
Flux<MappedType> aFluxMap = Library.createMappingToMappedType();
Flux.merge(aFluxMap, yourFlux);
The merge operator executes a merging of the MappedType objects from the two provided publisher sequences.
I'm looking for some help since I don't know how to optimize a process.
I have to invoke a service that returns a list with more than 500K elements (I don't know why, these services belongs to the client), per each element of the list, I have to invoke 2 more services and then save some attributes in our database, this last step is not the problem, but the entire process took between 1 and 2 seconds per element, so with this time is going to take like more of 100 hours to complete the process.
My approach is the following, I have my main method, inside this method I get the large list, then I use a parallelStream to iterate in the elements of the list and then I use a CompletableFuture to call the method that invokes the 2 services mentioned above. I've tried changing the parallelStream to stream and for-each , tried to split the main list into smaller lists and many other things but I don't see a better performance, I think the problem is the invocation of those 2 services but I want to try luck asking here.
I'm using java 11, spring, and for the invocation of the services I'm using RestTemplate, and this is my code:
public void updateDiscount() {
//List with 500k elements
var relationshipList = relationshipService.getLargeList();
//CompletableFuture to make the async calls to the method above
relationshipList.parallelStream().forEach(level1 -> {
CompletableFuture.runAsync(() -> relationshipService.asyncDiscountSave(level1));
});
}
//Second class
#Async("nameOfThePool")
public void asyncDiscountSave(ElementOfList element) {
//Logic to create request
//.........
var responseClients = anotherClass.getClients(element.getGroup1()) //get the first response with restTemplate
var responseProducts = anotherClass.getProducts(element.getGroup2())//get the second response with restTemplate
for (var client : responseClients) {
for (var product : responseProducts) {
//Here we just save some attributes of these objects on our DB
}
}
}
Thanks for the help.
UPDATE:
For this particular case, the only improvement that I can do is to pass a thread pool to the completable future, the problem is the response time of the services that I need to invoke.
I decided to follow a second approach and it took like 5 hours to complete, compared with the first approach this is acceptable.
As you haven't defined an executor you are using the default pool. Adding an executor allow you to create many threads as you needed and the server resources can manage
public void updateDiscount() {
Executor executor = Executors.newFixedThreadPool( 100 );//Define the number according to server resources performance
//List with 500k elements
var relationshipList = relationshipService.getLargeList();
//CompletableFuture to make the async calls to the method above
relationshipList.parallelStream().forEach(level1 -> {
CompletableFuture.runAsync(() -> relationshipService.asyncDiscountSave(level1), executor);
});
}
I'm working with a spring cloud gateway based project and my goal is to capture and log incoming and outgoing messages partially. Request logging must be done before request is passed to backend service and same policy applies to response. Implementation should be based on a filter. I have no control over when gateway subscribes to resulting flux.
In short, I would like to do following:
Capture up to x bytes of data from flux
Log captured data
Create a flux that contains both captured data and remaining data
So far I got this - and it seems to be working. I'd just like to know, if I missed something and/or if there's a better way to implement this. I'm sure someone else has been struggling with a similar problem:
Flux<Integer> body = Flux.range(1, 50).log(); // Simulate flow of data
ConnectableFlux<Integer> sharedBody = body.publish(1); // Content is already buffered - ideal prefetch would be 0
AtomicLong readCount = new AtomicLong(); // Counter
AtomicReference<Flux<Integer>> partiallyCachedFlux = new AtomicReference<>(); // A hack, not to be used in real world
Flux.from(sharedBody)
.takeUntil(s -> {
System.out.println("C:" + s);
return readCount.incrementAndGet() >= 10; // Store up to 10 elements
})
.collectList()
.subscribe(ints -> {
System.out.println("Collected:" + ints); // Log what we got
partiallyCachedFlux.set(
Flux.concat(Flux.fromIterable(ints).log(), sharedBody)
); // Create a flux that contains captured data and remaining data
});
sharedBody.connect();
Thread.sleep(1000); // Because I was lazy
partiallyCachedFlux.get()
.doOnEach(i -> { if (i.isOnNext()) System.out.println("P:" + i.get());})
.subscribe(); // Show that we have captured everything
The opposite of takeUntil is skipUntil. You could share the original flux into 2 flux, one of which takesUntil and the other skipsUntil. Your end result would simply be the Flux.merge of both flux.
Note that when externalizing state like this (AtomicInteger), you'll run into problems if the whole Flux is subscribed to multiple times. The way to work around that is to wrap everything into a Flux.defer, so that the external state is created within the lambda and thus specific to a given subscription.
I am writing in Java on the Vertx framework, and I have an architecture question regarding blocking code.
I have a JsonObject which consists of 10 objects, like so:
{
"system":"CD0",
"system":"CD1",
"system":"CD2",
"system":"CD3",
"system":"CD4",
"system":"CD5",
"system":"CD6",
"system":"CD7",
"system":"CD8",
"system":"CD9"
}
I also have a synchronous function which gets an object from the JsonObject, and consumes a SOAP web service, while sending the object to it.
the SOAP Web service gets the content (e.g. CD0), and after a few seconds returns an Enum.
I then want to take that enum value returned, and save it in some sort of data variable(like hash table).
What I ultimately want is a function that will iterate over all the JsonObject's objects, and for each one, run the blocking code, in parallel.
I want it to run in parallel so even if one of the calls to the function needs to wait 20 seconds, it won't stuck the other calls.
how can I do such a thing in vertx?
p.s: I will appreciate if you will correct mistakes I wrote.
Why not to use rxJava and "zip" separate calls? Vertx has great support for rxJava too. Assuming that you are calling 10 times same method with different String argument and returning another String you could do something like this:
private Single<String> callWs(String arg) {
return Single.fromCallable(() -> {
//DO CALL WS
return "yourResult";
});
}
and then just use it with some array of arguments:
String[] array = new String[10]; //get your arguments
List<Single<String>> wsCalls = new ArrayList<>();
for (String s : array) {
wsCalls.add(callWs(s));
}
Single.zip(wsCalls, r -> r).subscribe(allYourResults -> {
// do whatever you like with resutls
});
More about zip function and reactive programming in general: reactivex.io
I am not sure if i can put my question in the clearest fashion but i will try my best.
Lets say i am retrieving some information from a third party api. The retrieved information will be huge in size. To have a performance gain, instead of retrieving all the info in one go, i will be retrieving the info in a paged fashion (the api gives me that facility, basically an iterator). The return type is basically a list of objects.
My aim here is to process the information i have in hand(that includes comparing and storing in db and many other operations) while i get paged response on the request.
My question here to the expert community is , what data structure do you prefer in such case. Also does a framework like spring batch help you in getting performance gains in such cases.
I know the question is a bit vague, but i am looking for general ideas,tips and pointers.
In these cases, the data structure for me is java.util.concurrent.CompletionService.
For purposes of example, I'm going to assume a couple of additional constraints:
You want only one outstanding request to the remote server at a time
You want to process the results in order.
Here goes:
// a class that knows how to update the DB given a page of results
class DatabaseUpdater implements Callable { ... }
// a background thread to do the work
final CompletionService<Object> exec = new ExecutorCompletionService(
Executors.newSingleThreadExecutor());
// first call
List<Object> results = ThirdPartyAPI.getPage( ... );
// Start loading those results to DB on background thread
exec.submit(new DatabaseUpdater(results));
while( you need to ) {
// Another call to remote service
List<Object> results = ThirdPartyAPI.getPage( ... );
// wait for existing work to complete
exec.take();
// send more work to background thread
exec.submit(new DatabaseUpdater(results));
}
// wait for the last task to complete
exec.take();
This just a simple two-thread design. The first thread is responsible for getting data from the remote service and the second is responsible for writing to the database.
Any exceptions thrown by DatabaseUpdater will be propagated to the main thread when the result is taken (via exec.take()).
Good luck.
In terms of doing the actual parallelism, one very useful construct in Java is the ThreadPoolExecutor. A rough sketch of what that might look like is this:
public class YourApp {
class Processor implements Runnable {
Widget toProcess;
public Processor(Widget toProcess) {
this.toProcess = toProcess;
}
public void run() {
// commit the Widget to the DB, etc
}
}
public static void main(String[] args) {
ThreadPoolExecutor executor =
new ThreadPoolExecutor(1, 10, 30,
TimeUnit.SECONDS,
new LinkedBlockingDeque());
while(thereAreStillWidgets()) {
ArrayList<Widget> widgets = doExpensiveDatabaseCall();
for(Widget widget : widgets) {
Processor procesor = new Processor(widget);
executor.execute(processor);
}
}
}
}
But as I said in a comment: calls to an external API are expensive. It's very likely that the best strategy is to pull all the Widget objects down from the API in one call, and then process them in parallel once you've got them. Doing more API calls gives you the overhead of sending the data all the way from the server to you, every time -- it's probably best to pay that cost the fewest number of times that you can.
Also, keep in mind that if you're doing DB operations, it's possible that your DB doesn't allow for parallel writes, so you might get a slowdown there.