I'm new to Webflux and I'm trying to implement this scenario:
client ask for data
if data is already present in redis cache => return cached data
otherwise query remote service for data
I've written this code:
ReactiveRedisOperations<String, Foo> redisOps;
private Mono<Foo> getFoo(String k) {
return this.redisOperations.opsForValue().get(k)
.map(f -> this.logCache(k, f))
.switchIfEmpty(this.queryRemoteService(k));
}
private void logCache(String k, Foo f) {
this.logger.info("Foo # {} # {} present in cache. {}",
k,
null != f ? "" : "NOT",
null != f ? "" : "Querying remote");
}
private Mono<Foo> queryRemoteService(String k) {
this.logger.info("Querying remote service");
// query code
}
It prints:
"Querying remote service"
"Foo # test_key # present in cache"
How can I ensure that switchIfEmpty is called only if cached data is not present?
Edit
Accordingly to Michael Berry's answer I refactored my code like follows:
private Mono<Foo> getFoo(String k) {
this.logger.info("Trying to get cached {} data", k);
this.logger.info(this.redisOps.hasKey(k).block() ? "PRESENT" : "NOT present");
return this.redisOperations.opsForValue().get(k)
.switchIfEmpty(this.queryRemoteService(k));
}
private Mono<Foo> queryRemoteService(String k) {
this.logger.info("Querying remote service");
// query code
}
Now my output is this:
Trying to get cached test_key data
PRESENT
Querying provider
So it seems that is executed only one time, but still can't avoid switchIfEmpty is executed. I'm sure that redis contains data for that key
This line:
.map(f -> this.logCache(k, f))
...is rather odd, as you're not mapping anything to anything, you're instead performing a side effect (logging the value.) doOnNext() (rather than map()) would be a much more sensible choice here.
However, I digress, that's not the primary issue here.
How can I ensure that switchIfEmpty is called only if cached data is not present?
It's that way already, but your logging isn't doing what you think. The key concept likely causing you an issue here is that null can never be propagated through a reactive stream. If the stream is empty as it is in this example, then nothing is propagated at all.
Therefore, in your code, neither map() (nor doOnNext() if that were used) will be called, so your "present in cache" line won't be written to the logs (since there's no value to map, and no value to call a side effect against.) Checking whether the value is null in logCache() is therefore pointless - it will never be null.
As far as I see it, your log output here therefore must be the result of two invocations of getFoo():
The first was not present in the cache, so map() wasn't called, switchIfEmpty() switched, and "Querying remote service" was printed;
The second was present in the cache, so your "present in cache" line was printed, switchIfEmpty() wasn't called, and therefore "Querying remote service" wasn't printed.
To make your logging make sense, you should remove the conditional logic from logCache(), and add a "not present in cache" line to the queryRemoteService() method.
Related
Let's say that I have a method addVoteToSong like:
public Mono<Map<Song, VoteKind>> addVoteToSong(Principal principal, String songId, VoteKind voteKind) {
return
userRepository.findUserByUsername(principal.getName())
.doOnSuccess(song -> songRepository.findSongById(songId))
.doOnSuccess(vote -> voteRepository.add(Vote.builder().song()))
.//(the rest of the code)
}
I want to pass a result from the line:
userRepository.findUserByUsername(principal.getName())
and
.doOnSuccess(song -> songRepository.findSongById(songId))
to the built object in the line:
.doOnSuccess(vote -> voteRepository.add(Vote.builder().song(here result from findSongById).user(here result from findUserByUsername))
Here comes the question, is it possible to reuse previous API call result in the next doOnSuccess method or I should split find API calls at the same time, giving up on Reactor's cascading operations? On the internet, I have found examples with single save method without basing on the indirect result of the reactive stream and that's why question occurred. I will be grateful for suggestions on how to reach a goal.
Martin,
First of all, be aware that .doOnXXX are just callbacks that will be executed on some archived conditions. You should avoid putting a business logic inside of them.
Coming back to the question, the first idea that comes to my mind is to benefit from zip operator. So you have to put 2 publishers .findUserByUsername and .findSongById and combine the result using BiFunction. So you can try the following:
public Mono<Map<Song, VoteKind>> addVoteToSong(Principal principal, String songId, VoteKind voteKind) {
return Mono
.zip(
userRepository.findUserByUsername(principal.getName()),
songRepository.findSongById(songId),
(user, song) -> voteRepository.add(Vote.builder().song(song).user(user).build())
)
.flatMap(Function.identity())
// your code is here
}
spring-data-redis module contains RedisAtomicLong class.
In this class you can see
public boolean compareAndSet(long expect, long update) {
return generalOps.execute(new SessionCallback<Boolean>() {
#Override
#SuppressWarnings("unchecked")
public Boolean execute(RedisOperations operations) {
for (;;) {
operations.watch(Collections.singleton(key));
if (expect == get()) {
generalOps.multi();
set(update);
if (operations.exec() != null) {
return true;
}
}
{
return false;
}
}
}
});
}
My question is why it works?
generalOps.multi() starts transaction after get() is invoked. It means that there is possibility that two different thread (or even client) can change value and both of them will succeed.
Is operations.watch prevent it somehow? JavaDoc doesn't explain purpose of this method.
PS: Minor question: why for (;;)? There is always one iteration.
Q: Is operations.watch prevent it somehow?
YES.
Quoting from Redis documentation about transaction:
WATCH is used to provide a check-and-set (CAS) behavior to Redis transactions.
WATCHed keys are monitored in order to detect changes against them. If at least one watched key is modified before the EXEC command, the whole transaction aborts, and EXEC returns a Null reply to notify that the transaction failed.
You can learn more about Redis transaction from that documentation.
Q: why for (;;)? There is always one iteration.
It seems the code you've posted is very old. From Google's cache of this url, I saw the code you provided which is dated back to Oct 15th, 2012!
Latest codes look much different:
compareAndSet method
CompareAndSet class
Is operations.watch prevent it somehow?
YES. After watching a key, if the key has been modified before transaction finishes, EXEC will fail. So if EXEC successes, the value is guaranteed to be unchanged by others.
why for (;;)? There is always one iteration.
In your case, it seems the infinite loop is redundant.
However, if you want to implement a check-and-set operation to modify the value with the old value, the infinite loop is necessary. Check this example from redis doc:
WATCH mykey
val = GET mykey
val = val + 1
MULTI
SET mykey $val
EXEC
Since EXEC might fail, you need to retry the whole process in a loop until it successes.
RedisAtomicLong.compareAndSet implementation is not optimal since it requires 5 requests to Redis
Redisson - Redis Java client provides more efficient implementation.
org.redisson.RedissonAtomicLong#compareAndSetAsync method implemented using atomic EVAL-script:
"local currValue = redis.call('get', KEYS[1]); "
+ "if currValue == ARGV[1] "
+ "or (tonumber(ARGV[1]) == 0 and currValue == false) then "
+ "redis.call('set', KEYS[1], ARGV[2]); "
+ "return 1 "
+ "else "
+ "return 0 "
+ "end",
This script requires only single request to Redis.
Usage example:
RAtomicLong atomicLong = redisson.getAtomicLong("myAtomicLong");
atomicLong.compareAndSet(1L, 2L);
How can I return an object or list after using Blocking.get() method in Ratpack?
Blocking.get(()->
xRepository.findAvailable()).then(x->x.stream().findFirst().get());
Above line returns void - I want to be able to do something like below so that it returns the object in the then clause. I tried adding a return statement but doesn't work.
Object x = Blocking.get(()->
xRepository.findAvailable()).then(x->x.stream().findFirst().get());
You can use map to work with the value when it's available.
Blocking.get(() -> xRepository.findAvailable())
.map(x -> x.stream().findFirst().get())
.then(firstAvailable -> ctx.render("Here is the first available x " + firstAvailable))
Ratpack's Promise<T> does not provide blocking operation like Promise.get() that blocks current thread and returns a result. Instead you have to subscribe to the promise object. One of the methods you can use is Promise.then(Action<? super T> then) which allows you to specify and action that will be triggered when given value is available. In above example we use ctx.render() as an action triggered when value from blocking operation is ready, but you can do other things as well.
I'm trying to filter a list which might or might not be empty (or the item is not in the list). inboxData is filled by another observable:
private BehaviorSubject<InboxResponse> inboxData = BehaviorSubject.create();
public Observable<Item> getInboxItem(String id) {
return inboxData
.flatMap(response -> Observable.from(response.getData()))
.filter(item -> item.getId().equals(id))
.firstOrDefault(null);
}
In this case if response.getData() is empty firstOrDefault never runs. But why? It clearly says that it gives you back the default value if the preceeding observable emits nothing.
firstOrDefault emits the default if the stream completes without any items being passed through the observable. For your stream to complete the BehaviorSubject would need to signal completion. Since there is no indication that happens it would never realize it needs to send the default.
The solution is to move the filter and firstOrDefault to the inside of the flatMap so the end of the list provided by getData ends up completing the inner stream.
Note that if you're using RxJava2 as your tags suggest, null can never be an item in the stream, so passing it as default would cause an exception.
public Observable<Item> getInboxItem(String id) {
return inboxData
.flatMap(response -> Observable.from(response.getData()))
At this point, response.getData() returns null, right?
.filter(item -> item.getId().equals(id))
That means that item here is null. So item.getId() throws a NullPointerException. An error like that is immediately passed to the subscriber's onError handler. The firstOrDefault method will not even be called anymore, because the stream is immediately terminated.
.firstOrDefault(null);
}
JavaRDD<Text> tx= counts2.map(new Function<Object, Text>() {
#Override
public Text call(Object o) throws Exception {
// TODO Auto-generated method stub
if (o.getClass() == Dict.class) {
Dict rkd = (Dict) o;
return new Text(rkd.getId());
} else {
return null ;
}
}
});
tx.saveAsTextFile("/rowkey/Rowkey_new");
I am new to Spark, I want to save this file, but I got the Null exception. I don't want to use return new Text() to replace return null,because it will insert a blank line to my file. So how can I solve this problem?
Instead of putting an if condition in your map, you simply use that if condition to build a RDD filter. The Spark Quick Start is a good place to start. There is also a nice overview of other transformations and actions.
Basically your code can look as follows (if you are using Java 8):
counts2
.filter((o)->o instanceof Dict)
.map(o->new Text(((Dict)o).getId()))
.saveAsTextFile("/rowkey/Rowkey_new");
You had the intention to map one incoming record to either zero or one outgoing record. This cannot be done with a map. However, filter maps to zero or one records with incoming record matches outgoing record, and flatMap gives you some more flexibility by allowing to map to zero or more outgoing records of any type.
It is strange, but not inconceivable, you create non-Dict objects that are going to be filters out further downstream anyhow. Possibly you can consider to push your filter even further upstream to make sure you only create Dict instances. Without knowing the rest of your code, this is only a assumption of course, and is not part of your original question anyhow.