I've created a reactive flow at my controller Endpoint addEntry where one object inside should be created only once per request since it holds a state.
#Override
public Mono<FileResultDto> addEntry(final Flux<byte[]> body,
final String fileId) {
return keyVaultRepository.findByFiletId(fileId)
.switchIfEmpty(Mono.defer(() -> {
final KeyVault keyVault = KeyVault.of(fileId);
return keyVaultRepository.save(keyVault);
}))
.map(keyVault -> Mono
.just(encryption.createEncryption(keyVault.getKey(), ENCRYPT_MODE)) // createEncryption obj. that holds a state
.cache())
.map(encryption -> Flux
.from(body)
.map(bytes -> encryption
.share()
.block()
.update(bytes) // works with the state and changes it per byte[] going through this flux
)
)
.flatMap(flux -> persistenceService.addEntry(flux, fileId));
}
before I asked this question I used
encryption.block() which was failing.
I found this one and updated my code accordingly (added .share()).
The test itself is working. But I am wondering if this is the proper way to go to work with an object that should be created and used only once in the reactive flow, provided by
encryptionService.createEncryption(keyVault.getKey(), ENCRYPT_MODE)
Happy to hear your opinion
Mono.just is only a wrapper around a pre-computed value, so there is no need to cache or share it, because it is only just giving back a cached variable on subscription.
But, in your example, there is something I do not understand.
If we simplify / decompose it, it gives the following:
Mono<KeyVault> vault = keyVaultRepository.findByFiletId(fileId)
.switchIfEmpty(Mono.defer(() -> keyVaultRepository.save(KeyVault.of(fileId));
));
Mono<Mono<Encryption>> fileEncryption = vault
.map(it -> Mono.just(createEncryption(it.getKey)).cache()); // <1>
Mono<Flux<Encryption>> encryptedContent = fileEncryption.map(encryption -> Flux
.from(body)
.map(bytes -> encryption
.share()
.block()
.update(bytes))); // <2>
Mono<FileResultDto> file = encryptedContent.map(flux -> persistenceService.addEntry(flux, fileId));
Why are you trying to wrap your encryption object ? The result is already part of a reactive pipeline. Doing Mono.just() is redundant because you are already in a map operation, and doing cache() over just() is also redundant, because a "Mono.just" is essentially a permanent cache.
What does your "update(bytes)" method do ? Does it mutate the same object every time ? because if it does, you might have a problem here. Reactive streams cannot ensure thread-safety and proper ordering of actions on internal mutated states, that is out of its reach. You might bypass the problem by using scan operator, though.
Without additional details, I would start refactoring the code like this:
Mono<KeyVault> vault = keyVaultRepository.findByFileId(fileId)
.switchIfEmpty(Mono.defer(() -> keyVaultRepository.save(KeyVault.of(fileId));
Mono<Encryption> = vault.map(it -> createEncryption(it.getKey()));
Flux<Encryption> encryptedContent = fileEncryption
.flatMapMany(encryption -> body.scan(encryption, (it, block) -> it.update(block)));
Mono<FileResultDto> result = persistenceService.addEntry(encryptedContent, fileId);
Related
I'm working with a spring cloud gateway based project and my goal is to capture and log incoming and outgoing messages partially. Request logging must be done before request is passed to backend service and same policy applies to response. Implementation should be based on a filter. I have no control over when gateway subscribes to resulting flux.
In short, I would like to do following:
Capture up to x bytes of data from flux
Log captured data
Create a flux that contains both captured data and remaining data
So far I got this - and it seems to be working. I'd just like to know, if I missed something and/or if there's a better way to implement this. I'm sure someone else has been struggling with a similar problem:
Flux<Integer> body = Flux.range(1, 50).log(); // Simulate flow of data
ConnectableFlux<Integer> sharedBody = body.publish(1); // Content is already buffered - ideal prefetch would be 0
AtomicLong readCount = new AtomicLong(); // Counter
AtomicReference<Flux<Integer>> partiallyCachedFlux = new AtomicReference<>(); // A hack, not to be used in real world
Flux.from(sharedBody)
.takeUntil(s -> {
System.out.println("C:" + s);
return readCount.incrementAndGet() >= 10; // Store up to 10 elements
})
.collectList()
.subscribe(ints -> {
System.out.println("Collected:" + ints); // Log what we got
partiallyCachedFlux.set(
Flux.concat(Flux.fromIterable(ints).log(), sharedBody)
); // Create a flux that contains captured data and remaining data
});
sharedBody.connect();
Thread.sleep(1000); // Because I was lazy
partiallyCachedFlux.get()
.doOnEach(i -> { if (i.isOnNext()) System.out.println("P:" + i.get());})
.subscribe(); // Show that we have captured everything
The opposite of takeUntil is skipUntil. You could share the original flux into 2 flux, one of which takesUntil and the other skipsUntil. Your end result would simply be the Flux.merge of both flux.
Note that when externalizing state like this (AtomicInteger), you'll run into problems if the whole Flux is subscribed to multiple times. The way to work around that is to wrap everything into a Flux.defer, so that the external state is created within the lambda and thus specific to a given subscription.
Let's say that I have a method addVoteToSong like:
public Mono<Map<Song, VoteKind>> addVoteToSong(Principal principal, String songId, VoteKind voteKind) {
return
userRepository.findUserByUsername(principal.getName())
.doOnSuccess(song -> songRepository.findSongById(songId))
.doOnSuccess(vote -> voteRepository.add(Vote.builder().song()))
.//(the rest of the code)
}
I want to pass a result from the line:
userRepository.findUserByUsername(principal.getName())
and
.doOnSuccess(song -> songRepository.findSongById(songId))
to the built object in the line:
.doOnSuccess(vote -> voteRepository.add(Vote.builder().song(here result from findSongById).user(here result from findUserByUsername))
Here comes the question, is it possible to reuse previous API call result in the next doOnSuccess method or I should split find API calls at the same time, giving up on Reactor's cascading operations? On the internet, I have found examples with single save method without basing on the indirect result of the reactive stream and that's why question occurred. I will be grateful for suggestions on how to reach a goal.
Martin,
First of all, be aware that .doOnXXX are just callbacks that will be executed on some archived conditions. You should avoid putting a business logic inside of them.
Coming back to the question, the first idea that comes to my mind is to benefit from zip operator. So you have to put 2 publishers .findUserByUsername and .findSongById and combine the result using BiFunction. So you can try the following:
public Mono<Map<Song, VoteKind>> addVoteToSong(Principal principal, String songId, VoteKind voteKind) {
return Mono
.zip(
userRepository.findUserByUsername(principal.getName()),
songRepository.findSongById(songId),
(user, song) -> voteRepository.add(Vote.builder().song(song).user(user).build())
)
.flatMap(Function.identity())
// your code is here
}
In what way can we sync two asynchronous calls using RxJava? In the example below, the method contentService.listContents which is a API call must first finish before the processSchema method to take place for each schema.
schemaService.listSchema()
.toObservable()
.flatMapIterable(schemas -> {
schemas.forEach(schema -> {
// async call
contentService.listContents(schema.getName()).subscribe(contents -> {
doSomethingWithThe(contents);
});
});
// contentService.listContents` must complete first before
// processSchema should be called for each schema
return schemas;
}).subscribe(schema -> { processSchema(schema); },
error -> { Console.error(error.getMessage()); });
The problem with the code above the processSchema would not wait for the contentService.listContents since it is async not not synchronized with each other.
You have to use flatMap to process the schemas and since it is a list, you have to unroll it and flatMap again:
schemaService.listSchema()
.toObservable()
.flatMap(schemas ->
Observable.fromIterable(schemas)
.flatMap(schema ->
contentService.listContents(schema.getName())
.doOnNext(contents -> doSomethingWith(contents))
)
// probably you don't care about the inner contents
.ignoreElements()
// andThen will switch to this only when the sequence above completes
.andThen(Observable.just(schemas))
)
.subscribe(
schema -> processSchema(schema),
error -> Console.error(error.getMessage())
);
Note that you haven't defined the return types of the service calls so you may have to use flatMapSingle and doOnSuccess for example.
You are probably looking for flatMap.
From the docs
Continuations
Sometimes, when an item has become available, one would
like to perform some dependent computations on it. This is sometimes
called continuations and, depending on what should happen and what
types are involved, may involve various operators to accomplish.
Dependent
The most typical scenario is to given a value, invoke
another service, await and continue with its result:
service.apiCall()
.flatMap(value -> service.anotherApiCall(value))
.flatMap(next -> service.finalCall(next))
It is often the case also that later sequences would require values
from earlier mappings. This can be achieved by moving the outer
flatMap into the inner parts of the previous flatMap for example:
service.apiCall()
.flatMap(value ->
service.anotherApiCall(value)
.flatMap(next -> service.finalCallBoth(value, next))
)
I have 2 service endpoints in my application a and b (Both are "Singles"). The request to service b depends on the response of a. After the the response of b I need access to both responses in my subscriber. I plan to do the call like this:
services.a()
.flatMap(a -> services.b(a))
.subscribe(b ->
// Access a and b here
)
But in this way, I only can access the result of b in my subscriber. How can I pass also the response of a to it?
My first attempt was to use something like this:
// Note: Code will not compile... Just to show the concept
services.a()
.flatMap(
a -> Observable.combineLatest(
Observable.just(a)
services.b(a).toObservable()
Bifunction((a, b) -> {return Pair<ResponseA, ResponseB>(a, b)}))
)
.toSingle()
.subscribe(pair -> {
ResponseA a = pair.first();
ResponseB b = pair.second();
})
But as the use case gets a bit more complex, the code will evolve to an ugly monster.
You can use a variant of flatMap() with resultSelector, resultSelector method will gather both input (a) and output (result of Observable b) of the flatMap and you can combine them together to whatever you need. See here.
The problem I am facing is as follows:
I have two observables one is fetching data from network and the other from db. The second one might be empty but the lack of the first one is considered an error. Then if the result from network comes I need to compare it with the latest results from db ( if present ) and if they differ I want to store them ( if the db observable is empty I want to store network results anyway).
Is there any dedicated operator that handles a case like this?
So far I tried a solution with zipWith ( which is not working as expected if db is empty ), buffer ( which is working but is far from ideal ),
and I also tried flatmapping ( which requires additional casting in the subscriber ).
Below is the solution with buffer.
Observable.concat(ratesFromNetwork(), latestRatesFromDB())
.buffer(3000, 2)
.filter(buffer -> !(buffer.size() == 2 && !buffer.get(0).differentThan(buffer.get(1))))
.map(buffer -> buffer.get(0))
.subscribe(this::save,
(ex) -> System.out.println(ex.getMessage()),
() -> System.out.println("completed"));
If I modify latestRatesFromDb so that it is not returning Observable but and Optional instead the whole problem becomes trivial because I can filter using this result. It seams that there is no way to filter in an asynchronous way ( or did I miss something ?)
Okay, here is how I would go about writing this.
Firstly, whatever class has the differentThan function should be changed to override equals instead. Otherwise you can't use a lot of basic methods with these objects.
For the purpose of this example I wrote all the observables using the Integer class as my type parameter. I then use a scheduler to write two mock methods:
static Observable<Integer> ratesFromNetwork(Scheduler scheduler) {
return Observable.<Integer>create(sub -> {
sub.onNext(2);
sub.onCompleted();
}).delay(99, TimeUnit.MILLISECONDS, scheduler);
}
static Observable<Integer> latestRatesFromDB(Scheduler scheduler) {
return Observable.<Integer>create(sub -> {
sub.onNext(1);
sub.onCompleted();
}).delay(99, TimeUnit.MILLISECONDS, scheduler);
}
As you can see both are similar, however, they will emit different values.
lack of the first one is considered an error
The best way to achieve this is to use a timeout. You can log the error immediately here and continue:
final Observable<Integer> networkRate = ratesFromNetwork(scheduler)
.timeout(networkTimeOut, TimeUnit.MILLISECONDS, scheduler)
.doOnError(e -> System.err.println("Failed to get rates from network."));
When the timeout fails an error will be thrown by rx. doOnError will give you a better idea of where this error started and let it propagate through the rest of the sequence.
The second one might be empty
In this case I would do a similar strategy, however, do not let the error propagate by using the method onErrorResumeNext. Now you can make sure the observable emits at least one value by using firstOrDefault. In this method use some dummy value that you expect to never match with the network results.
final Observable<Integer> databaseRate = latestRatesFromDB(scheduler)
.timeout(databaseTimeOut, TimeUnit.MILLISECONDS, scheduler)
.doOnError(e -> System.err.println("Failed to get rates from database"))
.onErrorResumeNext(Observable.empty())
.firstOrDefault(-1);
Now by using the distinct method you can grab a value only when it is different than the one that came before it (which is why you need to override equals).
databaseRate.concatWith(networkRate).distinct().skip(1)
.subscribe(i -> System.out.println("Updating to " + i),
System.err::println,
() -> System.out.println("completed"));
Here the database rate was placed before the network rate to take advantage of distinct. a skip is then added to always ignore the database rate value.
Complete Code:
final long networkTimeOut = 100;
final long databaseTimeOut = 100;
final TestScheduler scheduler = new TestScheduler();
final Observable<Integer> networkRate = ratesFromNetwork(scheduler)
.timeout(networkTimeOut, TimeUnit.MILLISECONDS, scheduler)
.doOnError(e -> System.err.println("Failed to get rates from network."));
final Observable<Integer> databaseRate = latestRatesFromDB(scheduler)
.timeout(databaseTimeOut, TimeUnit.MILLISECONDS, scheduler)
.doOnError(e -> System.err.println("Failed to get rates from database"))
.onErrorResumeNext(Observable.empty())
.firstOrDefault(-1);
databaseRate.concatWith(networkRate).distinct().skip(1)
.subscribe(i -> System.out.println("Updating to " + i),
System.err::println,
() -> System.out.println("completed"));
scheduler.advanceTimeBy(200, TimeUnit.MILLISECONDS);
When networkTimeOut and databaseTimeOut are greater than 100 it prints:
Updating to 2
completed
When networkTimeOut is less than 100 it prints:
Failed to get rates from network.
java.util.concurrent.TimeoutException
When databaseTimeOut is less than 100 it prints:
Failed to get rates from database
Updating to 2
completed
And if you modify latestRatesFromDB and ratesFromNetwork to return the same value, it simply prints:
completed
And if you don't care about forcing timeouts or logging then it boils down to:
latestRatesFromDB().firstOrDefault(dummyValue)
.concatWith(ratesFromNetwork())
.distinct().skip(1)
.subscribe(this::save,
System.err::println,
() -> System.out.println("completed"));