I have this Spring WebFlux controller:
#RestController
public class Controller
{
#PostMapping("/doStuff")
public Mono<Response> doStuff(#RequestBody Mono<Request> request)
{
...
}
}
Now, say I wanted to relate separate requests coming to this controller from different clients to group processing based on some property of the Request object.
Take 1:
#PostMapping("/doStuff")
public Mono<Response> doStuff(#RequestBody Mono<Request> request)
{
return request.flux()
.groupBy(r -> r.someProperty())
.flatMap(gf -> gf.map(r -> doStuff(r)));
}
This will not work, because every call will get its own instance of the stream. The whole flux() call doesn't make sense, there will always ever be one Request object going through the stream even if there's many of those streams fired at the same time as a result of simultaneous calls coming from clients. What I need, I gather, is some part of the stream that is shared between all requests where I could do my grouping, which led me to this slightly over engineered code
Take 2:
private AtomicReference<FluxSink<Request>> sink = new AtomicReference<>();
private Flux<Response> serializingStream;
public Controller()
{
this.serializingStream =
Flux.<Request>create(fluxSink -> sink.set(fluxSink), ERROR)
.groupBy(r -> r.someProperty())
.flatMap(gf -> gf.map(r -> doStuff(r)));
.publish()
.autoConnect();
this.serializingStream.subscribe().dispose(); //dummy subscription to set the sink;
}
#PostMapping("/doStuff")
public Mono<Response> doStuff(#RequestBody Request request)
{
req.setReqId(UUID.randomUUID().toString());
return
serializingStream
.doOnSubscribe(__ -> sink.get().next(req))
.filter(resp -> resp.getReqId().equals(req.getReqId()))
.take(1)
.single();
}
And this kind of works, though it looks like I am doing things I shouldn't (or at least they don't feel right), like leaking the FluxSink and then injecting a value through it while subscribing, adding a request ID so that I can then filter the right response. Also, if error happens in the serializingStream then it breakes everything for everyone, but I guess I could try to isolate the errors to keep things going.
The question is, is there a better way of doing this that doesn't feel like an open heart surgery.
Also, related question for a similar scenario. I was thinking about using Akka Persistence to implement event sourcing and have it trigerred from inside that Reactor stream. I was reading about Akka Streams that allow to wrap an Actor and then there's some ways of converting that into something that can be hooked up with Reactor (aka Publisher or Subscriber), but then if every requests gets it's own stream, I am effectively loosing back pressure and am risking OOME because of flooding the Persistent Actor's mailbox, so I guess that problem falls in to the same category like the one I described above.
Related
While working with Spring Webflux, I'm trying to insert some data in the realm object server which interacts with Java apps via a Rest API. So basically I have a set of students, who have a set of subjects and my objective is to persist those subjects in a non-blocking manner. So I use a microservice exposed via a rest endpoint which provides me with a Flux of student roll numbers, and for that flux, I use another microservice exposed via a rest endpoint that gets me the Flux of subjects, and for each of these subjects, I want to persist them in the realm server via another rest endpoint. I wanted to make this all very nonblocking which is why I wanted my code to look like this.
void foo() {
studentService.getAllRollnumbers().flatMap(rollnumber -> {
return subjectDirectory.getAllSubjects().map(subject -> {
return dbService.addSubject(subject);
})
});
}
But this doesn't work for some reason. But once I call blocks on the things, they get into place, something like this.
Flux<Done> foo() {
List<Integer> rollNumbers = studentService.getAllRollnumbers().collectList().block();
rollNumbers.forEach(rollNumber -> {
List<Subject> subjects = subjectDirectory.getAllSubjects().collectList().block();
subjects.forEach(subject -> {dbService.addSubject(subject).block();});
});
return Flux.just(new NotUsed());
}
getAllRollnumbers() returns a flux of integers.
getAllSubjects() returns a flux of subject.
and addSubject() returns a Mono of DBResponse pojo.
What I can understand is that the thread executing this function is getting expired before much of it gets triggerred. Please help me work this code in an async non blocking manner.
You are not subscribing to the Publisher at all in the first instance that is why it is not executing. You can do this:
studentService.getAllRollnumbers().flatMap(rollnumber -> {
return subjectDirectory.getAllSubjects().map(subject -> {
return dbService.addSubject(subject);
})
}).subscribe();
However it is usually better to let the framework take care of the subscription, but without seeing the rest of the code I can't advise.
I have a request that is rather simple to formulate, but I cannot pull it of without leaking resources.
I want to return a response of type application/stream+json, featuring news events someone posted. I do not want to use Websockets, not because I don't like them, I just want to know how to do it with a stream.
For this I need to return a Flux<News> from my restcontroller, that is continuously fed with news, once someone posts any.
My attempt for this was creating a Publisher:
public class UpdatePublisher<T> implements Publisher<T> {
private List<Subscriber<? super T>> subscribers = new ArrayList<>();
#Override
public void subscribe(Subscriber<? super T> s) {
subscribers.add(s);
}
public void pushUpdate(T message) {
subscribers.forEach(s -> s.onNext(message));
}
}
And a simple News Object:
public class News {
String message;
// Constructor, getters, some properties omitted for readability...
}
And endpoints to publish news respectively get the stream of news
// ...
private UpdatePublisher<String> updatePublisher = new UpdatePublisher<>();
#GetMapping(value = "/news/ticker", produces = "application/stream+json")
public Flux<News> getUpdateStream() {
return Flux.from(updatePublisher).map(News::new);
}
#PutMapping("/news")
public void putNews(#RequestBody News news) {
updatePublisher.pushUpdate(news.getMessage());
}
This WORKS, but I cannot unsubscribe, or access any given subscription again - so once a client disconnects, the updatePublisher will just continue to push onto a growing number of dead channels - as I have no way to call the onCompleted() handler on the subscriptions.
TL;DL:
Can one push messages onto a possible endless Flux from a different thread and still terminate the Flux on demand without relying on a reset by peer exception or something along those lines?
You should never try to implement yourself the Publisher interface, as it boils down to getting the reactive streams implementation right. This is exactly the issue you're facing here.
Instead you should use one of the generator operators provided by Reactor itself (this is actually a Reactor question, nothing specific to Spring WebFlux).
In this case, Flux.create or Flux.push are probably the best candidates, given your code uses some type of event listener to push events down the stream. See the reactor project reference documentation on that.
Without more details, it's hard to give you a concrete code sample that solves your problem. Here are a few pointers though:
you might want to .share() the stream of events for all subscribers if you'd like some multicast-like communication pattern
pay attention to the push/pull/push+pull model that you'd like to have here; how is the backpressure supposed to work here? What if we produce more events that the subscribers can handle?
this model would only work on a single application instance. If you'd like this to work on multiple application instances, you might want to look into messaging patterns using a broker
I'm trying to use this interesting repository method :
#Tailable
Flux<Movie> findWithTailableCursorBy();
by expose it in a controller,
to stream new saved docs in a capped collection:
This is a DataAppInitializr :
#EventListener(ApplicationReadyEvent.class)
public void run(ApplicationReadyEvent evt) {
operations.collectionExists(Movie.class)
.flatMap(exists -> exists ? operations.dropCollection(Movie.class) : Mono.just(exists))
.then(operations.createCollection(Movie.class, CollectionOptions.empty()
.size(256 * 256)
.maxDocuments(10)
.capped()))
.thenMany(operations.insertAll(Flux.just("Jeyda", "Kaf Efrit").map(title-> new Movie(title)).collectList()))
.subscribe();
}
This is the controller method :
#GetMapping(value = "/tail", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
Flux<Movie> allTail() {
return movieRepository.findWithTailableCursorBy();
}
I got no exception,
I’m just getting a white page in the browser and no stream of new docs. Am I missing a step ?
Thank you in advance!
There are two aspects in your question that do not fit what you want to achieve:
Your code contains blocking bits: block(). Do not call .block() in initializers and event handlers during startup or when receiving events triggered by reactive infrastructure. Blocking is the easiest way to disrupt any functionality and makes your application defunct.
Browsers aren't the ideal tool to consume streams with a page view. Rather use cURL.
Besides that, you seem to have a mismatch between Flux<Person> and Flux<Movie>.
The issue comes from SecurityWebFilterChain from the spring-security-webflux. I should contact concerned people to notify them.
Thank you for your support!
I am rather new to RxJava and - as so many others - am trying to get my head around exception handling. I read quite a few post online (e.g. this discussion here how to handle exceptions thrown by observer's onNext) and think that I get the basic idea of the concepts.
In the above mentioned discussion, one of the posters says, that when an exception is thrown in a subscriber, RxJava does the following:
Implement generic handling to log the failure and stop sending it events
(of any kind) and clean up any resources due to that subscriber and carry
on with any remaining subscriptions.
This is also more or less what I see, the only thing I have problems with is the "clean up any ressources" bit. To make that clear, let's assume the following example:
I want to create an Observable that listens to an async event source (e.g. a JMS queue) and onNext()s on every received message. So in (pseudo-) code I would do something similar to this:
Observable<String> observable = Observable.create( s -> {
createConnectionToBroker();
getConsumer().setMessageListener(message -> s.onNext(transform(message)));
s.setDisposable(new Disposable() {
public void dispose() {
tearDownBrokerConnection();
}
});
});
Since I want to reuse the message listener for many subscribers / observers, I do not directly subscribe at the created Observable, but make use of the publish().refCount() team instead. Something similar to this:
Observable<String> observableToSubscribeTo = observable.publish().refCount();
Disposable d1 = observableToSubscribeTo.subscribe(s -> ...);
Disposable d2 = observableToSubscribeTo.subscribe(s -> ...);
This all works as expected. The code connects to JMS only when the first subscription is established, and the connection to the broker is closed when the last observer was dispose()d.
However, when a subscriber throws an exception when it is onNext()ed, things seem to get messy. As expected, the observer that threw is nuked, and whenever a new event is published, it won't be notified anymore. My problem appears that when all the remaining subscribers are dispose()d, the Observable that maintains the connection to the message broker is no longer notified. It looks to me as if the subscriber that threw the exception is in some sort of zombie state. It is ignored when it comes to event distribution, but it somehow prevents the root Observable to get notified when the last subscriber is dispose()d.
I understand that RxJava expects the observers to make sure to not throw but rather handle an eventual exception properly. Unfortunately, in the case where I want to provide a library that returns an Observable to the caller, I have no control over my subscribers whatsoever. This means, I would never be able to protect my library against stupid observers.
So, I am asking myself: am I missing something here? Is there really no chance to properly cleanup when a subscriber throws? Is this a bug or is it just me not understanding the library?
Any insights greatly appreciated!
If you could show some unit tests that demonstrates the problem (without the need for JMS) that would be great.
Also, onNext in RxJava 2 should never throw; if it does it is an undefined behavior. If you don't trust your consumers, you can have an end-observable transformer that does safeSubscribe instead of the plain subscribe that adds protecting against misbehaving downstream:
.compose(o -> v -> o.safeSubscribe(v))
or
.compose(new ObservableTransformer<T>() {
#Override public Observable<T> apply(final Observable<T> source) {
return new Observable<T>() {
#Override public void subscribeActual(Observer<? super T> observer) {
source.safeSubscribe(observer);
}
};
}
})
First of all, sorry for that abstract title. The idea is more simple with an example. Let say I have some values in a list L. I want to build parameters Request for a service, then call this service and collect all the Responses.
Currently, I'm using this kind of code structure :
private final List<String> bodies = ImmutableList.of(
"1", "2", "3", "4"
);
#Test
public void BasicRequestResponseStreamToList() {
final List<Request> requests = bodies.stream()
.map(Request::new)
.collect(Collectors.toList());
final List<Response> responses = requests.stream()
.map(service::send)
.collect(Collectors.toList());
commonAssertions(requests, responses);
}
However, I find the need of two stream not efficient considering the last request has to be built before the first one can be sent. I would like to do something like :
#Test
public void StatefulMapperStreamRequestResponseToList() {
final List<Request> requests = new ArrayList<>();
final List<Response> responses = bodies.stream()
.map(Request::new)
.map(x -> {
requests.add(x);
return service.send(x);
})
.collect(Collectors.toList());
commonAssertions(requests, responses);
}
However, I feel guilty to use such a "Hack" to the mapping semantic. However it's the only way I've found to build 2 Correlated list with lazy loading. The first solution doesn't interest me because it has to wait to build all the request before sending them.
I would love to achieve something like a wiretap in EIP. http://camel.apache.org/wire-tap.html
I would gladly have your thoughts about a more elegant manner than modifying the semantic of the map method to achieve this.
If it helps, you can find the source here : http://tinyurl.com/hojkdzu
Using .peek() while requires less changes in your code, is actually quite dirty solution. You need it, because you have a design flaw in your original code. You have "parallel data structures" (probably the term is not very good): the first element in the requests list corresponds to the first element in the responses list, and so on. When you have such situation, consider creating a new PoJo class instead. Something like this:
public class RequestAndResponse { // You may think up a better name
public final Request req; // use getters if you don't like public final fields
public final Response resp;
public RequestAndResponse(Request req, Response resp) {
this.req = req;
this.resp = resp;
}
}
Now your problem magically disappears. You can write:
List<RequestAndResponse> reqresp = bodies.stream()
.map(Request::new)
.map(req -> new RequestAndResponse(req, service.send(req)))
.collect(Collectors.toList());
commonAssertions(reqresp);
You will need to change commonAssertions method after that, but I'm pretty sure it will become simpler. Also you may find that some methods in your code use request and response together, so it's quite natural to make them as methods in RequestAndResponse class.
Not sure what you really want to achieve here. But if what you want is to collect requests "as you go" then you can use peek():
final List<Request> requests = new ArrayList<>();
final List<Response> responses = bodies.stream()
.map(Request::new)
.peek(requests::add)
.map(service::send)
.collect(Collectors.toList());
commonAssertions(requests, responses);
.peek() is pretty much what you mean in the subject of your post; it takes a Consumer, can occur at any steps in the pipeline, and here the consumer just saves the "intermediate states" into a list.
BUT... the javadoc of peek() specifically mentions this:
For parallel stream pipelines, the action may be called at whatever time and in whatever thread the element is made available by the upstream operation. If the action modifies shared state, it is responsible for providing the required synchronization.
So, well, be careful, I guess...