Develop Reference

Debounce similar requests with reactor-grpc

In order to offload my database, I would like to debounce similar requests in a gRPC service (say for instance that they share the same id part of the request) that serves an API which does not have strong requirements in terms of latency. I know how to do that with vanilla gRPC but I am sure what kind of API of Mono I can use.
The API calling directly the db looks like this:
public Mono<Blob> getBlob(
Mono<MyRequest> request) {
return request.
map(reader.getBlob(request.getId()));
I have a feeling I should use delaySubscription but then it does not seem that groupBy is part of the Mono API that gRPC services handle.

It's perfeclty ok to detect duplicates not using reactive operators:
// Guava cache as example.
private final Cache<String, Boolean> duplicatesCache = CacheBuilder.newBuilder()
.expireAfterWrite(Duration.ofMinutes(1))
.build();
public Mono<Blob> getBlob(Mono<MyRequest> request) {
return request.map(req -> {
var id = req.getId();
var cacheKey = extractSharedIdPart(id);
if (duplicatesCache.getIfPresent(cacheKey) == null) {
duplicatesCache.put(cacheKey, true);
return reader.getBlob(id);
} else {
return POISON_PILL; // Any object that represents debounce hit.
// Or use flatMap() + Mono.error() instead.
}
});
}
If for some reason you absolutely want to use reactive operators, then first you need to convert incoming grpc requests into Flux. This can be achieved using thirdparty libs like salesforce/reactive-grpc or directly:
class MyService extends MyServiceGrpc.MyServiceImplBase {
private FluxSink<Tuple2<MyRequest, StreamObserver<MyResponse>>> sink;
private Flux<Tuple2<MyRequest, StreamObserver<MyResponse>>> flux;
MyService() {
flux = Flux.create(sink -> this.sink = sink);
}
#Override
public void handleRequest(MyRequest request, StreamObserver<MyResponse> responseObserver) {
sink.next(Tuples.of(request, responseObserver));
}
Flux<Tuple2<MyRequest, StreamObserver<MyResponse>>> getFlux() {
return flux;
}
}
Next you subscribe to this flux and use operators you like:
public static void main(String[] args) {
var mySvc = new MyService();
var server = ServerBuilder.forPort(DEFAULT_PORT)
.addService(mySvc)
.build();
server.start();
mySvc.getFlux()
.groupBy(...your grouping logic...)
.flatMap(group -> {
return group.sampleTimeout(...your debounce logic...);
})
.flatMap(...your handling logic...)
.subscribe();
}
But beware of using groupBy with lots of distinct shared id parts:
The groups need to be drained and consumed downstream for groupBy to work correctly. Notably when the criteria produces a large amount of groups, it can lead to hanging if the groups are not suitably consumed downstream (eg. due to a flatMap with a maxConcurrency parameter that is set too low).

Infinite flux of finite fluxes. Or: how to reconnect to a websocket steam without completing the flux

I'm not sure if what I'm trying to do is anywhere near best practice. If not, I'd like to know!
I'm building a system that reacts to external events posted on websockets. After authentication I can request events to be emitted and listen for them on a single flux. I can attach actuators to that flux to do stuff (think home automation, button presses turn on lights).
A problem occurs when the websocket connection drops: The flux completes and I have to re-attach all the listening actuators to a new flux on a new connection. I thought it might be easier to have an infinite flux that just starts emitting again when there is a new connection with new events (auth and requesting the events has to be done again during startup of new connection of course).
I came up with a solution that does what I want it to do, but it feels like a big hack:
private Flux<String> finite() {
return Flux.interval(Duration.ofMillis(10)).handle((i, sink) -> {
if (i == 10) {
sink.complete();
} else {
sink.next(i);
}
}).map(it -> "hi :" + it).take(100);
}
public static class Holder {
#Setter
FluxSink<Object> sink;
public void next(EmitterProcessor<String> output) {
sink.next(output);
}
}
EmitterProcessor<String> output;
Holder holder;
#DisplayName("Should create infinite flux of finite fluxes")
#Test
public void infiniteFlux() throws InterruptedException {
AtomicInteger count1 = new AtomicInteger(0);
AtomicInteger count2 = new AtomicInteger(0);
holder = new Holder();
ConnectableFlux flux = Flux.create(fluxSink -> holder.setSink(fluxSink)).map(it -> (Flux) it).flatMap(it -> it).publish();
flux.connect();
flux.subscribe((it) -> count1.incrementAndGet());
Thread.sleep(150);
output = EmitterProcessor.create();
finite().subscribeWith(output);
holder.next(output);
Thread.sleep(150);
flux.subscribe((it) -> count2.incrementAndGet());
output = EmitterProcessor.create();
finite().subscribeWith(output);
holder.next(output);
Thread.sleep(150);
assertEquals(20, count1.get());
assertEquals(10, count2.get());
}
Can this maybe be done in a way with a processor in stead? Or should I forget about it and reconnect everything on completion of the websocket connection?
This is my current websocket implementation. I don't really like that I need to "know" all the types of events when starting to listen, but that is because I'm still not 100% sure how to handle sending and receiving with websockets
public void startListening(Flux<HaEventRequest> eventRequests) {
Mono<String> login = Mono.just(loginPayload());
Flux<String> subscribe = eventRequests
.doOnNext(req -> log.info("Registering: " + req.getEventType()))
.map(Json::write);
Flux<String> input = login.concatWith(subscribe);
EmitterProcessor<HaEventResponse> output = EmitterProcessor.create();
Mono<Void> sessionMono = client.execute(URI.create(wsUrl), session -> session.send(input.map(session::textMessage))
.thenMany(session.receive()
.map(WebSocketMessage::getPayloadAsText)
.map(message -> Json.read(message, HaEventResponse.class))
.filter(eventResponse -> eventResponse.getEvent() != null)
.subscribeWith(output)
.then())
.then());
eventFlux = output.doOnSubscribe(s -> sessionMono.subscribe())
.onErrorContinue((throwable, o) -> log.error("Error occured during web request. Dumping it: ", throwable));
}
public Flux<HaEventResponse> streamEvents(String eventType) {
return eventFlux.filter(eventResponse -> eventResponse.getEvent()
.getEventType()
.equals(eventType));
}

Convert Observable of List to List of Observables and merge in RxJava

I'm learning Java with Android by creating Hacker News reader app.
What I'm trying to do is:
Send a request to /topstories, return Observable<List<int>>, emit when
request finishes.
Map each storyId to Observable<Story>
Merge Observables into one entity, which emits List<Story>, when all requests finishes.
And to the code:
private Observable<Story> getStoryById(int articleId) {
BehaviorSubject<Story> subject = BehaviorSubject.create();
// calls subject.onNext on success
JsonObjectRequest request = createStoryRequest(articleId, subject);
requestQueue.add(request);
return subject;
}
public Observable<ArrayList<Story>> getTopStories(int amount) {
Observable<ArrayList<Integer>> topStoryIds = (storyIdCache == null)
? fetchTopIds()
: Observable.just(storyIdCache);
return topStoryIds
.flatMap(id -> getStoryById(id))
// some magic here
}
Then we would use this like:
getTopStories(20)
.subscribe(stories -> ...)

You can try something like that
Observable<List<Integers>> ids = getIdsObservable();
Single<List<Story>> listSingle =
ids.flatMapIterable(ids -> ids)
.flatMap(id -> getStoryById(id)).toList();
Then you can subscribe to that Single to get the List<Story>

Please have a look at my solution. I changed your interface to return a Single for getStoryById(), because it should only return one value. After that, I created a for each Story a Single request and subscribed to all of them with Single.zip. Zip will execute given lambda, when all Singles are finished. On drawback is, that all requestes will be fired at once. If you do not want this, I will update my post. Please take into considerations that #elmorabea solution will also subscribe to the first 128 elements (BUFFER_SIZE = Math.max(1, Integer.getInteger("rx2.buffer-size", 128));), and to the next element when one finishes.
#Test
void name() {
Api api = mock(Api.class);
when(api.getTopStories()).thenReturn(Flowable.just(Arrays.asList(new Story(1), new Story(2))));
when(api.getStoryById(eq(1))).thenReturn(Single.just(new Story(888)));
when(api.getStoryById(eq(2))).thenReturn(Single.just(new Story(888)));
Flowable<List<Story>> listFlowable =
api.getTopStories()
.flatMapSingle(
stories -> {
List<Single<Story>> collect =
stories
.stream()
.map(story -> api.getStoryById(story.id))
.collect(Collectors.toList());
// possibly not the best idea to subscribe to all singles at the same time
Single<List<Story>> zip =
Single.zip(
collect,
objects -> {
return Arrays.stream(objects)
.map(o -> (Story) o)
.collect(Collectors.toList());
});
return zip;
});
TestSubscriber<List<Story>> listTestSubscriber =
listFlowable.test().assertComplete().assertValueCount(1).assertNoErrors();
List<List<Story>> values = listTestSubscriber.values();
List<Story> stories = values.get(0);
assertThat(stories.size()).isEqualTo(2);
assertThat(stories.get(0).id).isEqualTo(888);
assertThat(stories.get(1).id).isEqualTo(888);
}
interface Api {
Flowable<List<Story>> getTopStories();
Single<Story> getStoryById(int id);
}
static class Story {
private final int id;
Story(int id) {
this.id = id;
}
}

Prevent race condition with RxJava asynchronous thread implementation

I am using spring framework StringRedisTemplate to update an entry which happen with multiple threads.
public void processSubmission(final String key, final Map<String, String> submissionDTO) {
final String hashKey = String.valueOf(Hashing.MURMUR_HASH.hash(key));
this.stringRedisTemplate.expire(key, 60, TimeUnit.MINUTES);
final HashOperations<String, String, String> ops = this.stringRedisTemplate.opsForHash();
Map<String, String> data = findByKey(key);
String json;
if (data != null) {
data.putAll(submissionDTO);
json = convertSubmission(data);
} else {
json = convertSubmission(submissionDTO);
}
ops.put(key, hashKey, json);
}
In this json entry looks below,
key (assignmentId) -> value (submissionId, status)
As seen in code, before update the cache entry, I fetch current entry and add the new entry and put them all. But since this operation can be do in multiple threads, there can be situation of race condition leads to data lost. I could synchronize above method, but then it will be a bottle neck for the parallel processing power of RxJava implementation where processSubmission method is call via RxJava on two asynchronous threads.
class ProcessSubmission{
#Override
public Observable<Boolean> processSubmissionSet1(List<Submission> submissionList, HttpHeaders requestHeaders) {
return Observable.create(observer -> {
for (final Submission submission : submissionList) {
//Cache entry insert method invoke via this call
final Boolean status = processSubmissionExecutor.processSubmission(submission, requestHeaders);
observer.onNext(status);
}
observer.onCompleted();
});
}
#Override
public Observable<Boolean> processSubmissionSet2(List<Submission> submissionList, HttpHeaders requestHeaders) {
return Observable.create(observer -> {
for (final Submission submission : submissionList) {
//Cache entry insert method invoke via this call
final Boolean status = processSubmissionExecutor.processSubmission(submission, requestHeaders);
observer.onNext(status);
}
observer.onCompleted();
});
}
}
Above will call from below service API.
class MyService{
public void handleSubmissions(){
final Observable<Boolean> statusObser1 = processSubmission.processSubmissionSet1(subListDtos.get(0), requestHeaders)
.subscribeOn(Schedulers.newThread());
final Observable<Boolean> statusObser2 = processSubmission.processSubmissionSet2(subListDtos.get(1), requestHeaders)
.subscribeOn(Schedulers.newThread());
statusObser1.subscribe();
statusObser2.subscribe();
}
}
So handleSubmissions is calling with multiple threads per assignment id. But then per main thread is create and call two reactive java threads and process the submission list associate with each assignment.
What would be the best approach I could prevent redis entry race condition, while keep the performance of the RxJava implementation? Is there a way I could do this redis operation more efficient way?

It looks like you're only using the ops variable to do a put operation at the end, and you could isolate that point which is where you need to synchronize.
In the short research that I did, I couldn't find if HashOperations is not already thread-safe).
But an example of how you could just isolate the part you're concerned about is to do something like:
public void processSubmission(final String key, final Map<String, String> submissionDTO) {
final String hashKey = String.valueOf(Hashing.MURMUR_HASH.hash(key));
this.stringRedisTemplate.expire(key, 60, TimeUnit.MINUTES);
Map<String, String> data = findByKey(key);
String json;
if (data != null) {
data.putAll(submissionDTO);
json = convertSubmission(data);
} else {
json = convertSubmission(submissionDTO);
}
putThreadSafeValue(key, hashKey, json);
}
And have a method that is synchronized just for the put operation:
private synchronized void putThreadSafeValue(key, hashKey, json) {
final HashOperations<String, String, String> ops = this.stringRedisTemplate.opsForHash();
ops.put(key, hashKey, json);
}
There are a number of ways to do this, but it looks like you could restrict the thread contention down to that put operation.

Rxjava2 + Retrofit2 + Android. Best way to do hundreds of network calls

I have an app. I have a big button that allows the user to sync all their data at once to the cloud. A re-sync feature that allows them to send all their data again. (300+ entries)
I am using RXjava2 and retrofit2. I have my unit test working with a single call. However I need to make N network calls.
What I want to avoid is having the observable call the next item in a queue. I am at the point where I need to implement my runnable. I have seen a bit about Maps but I have not seen anyone use it as a queue. Also I want to avoid having one item fail and it report back as ALL items fail, like the Zip feature would do. Should I just do the nasty manager class that keeps track of a queue? Or is there a cleaner way to send several hundred items?
NOTE: SOLUTION CANNOT DEPEND ON JAVA8 / LAMBDAS. That has proved to be way more work than is justified.
Note all items are the same object.
#Test
public void test_Upload() {
TestSubscriber<Record> testSubscriber = new TestSubscriber<>();
ClientSecureDataToolKit clientSecureDataToolKit = ClientSecureDataToolKit.getClientSecureDataKit();
clientSecureDataToolKit.putUserDataToSDK(mPayloadSecureDataToolKit).subscribe(testSubscriber);
testSubscriber.awaitTerminalEvent();
testSubscriber.assertNoErrors();
testSubscriber.assertValueCount(1);
testSubscriber.assertCompleted();
}
My helper to gather and send all my items
public class SecureDataToolKitHelper {
private final static String TAG = "SecureDataToolKitHelper";
private final static SimpleDateFormat timeStampSimpleDateFormat =
new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
public static void uploadAll(Context context, RuntimeExceptionDao<EventModel, UUID> eventDao) {
List<EventModel> eventModels = eventDao.queryForAll();
QueryBuilder<EventModel, UUID> eventsQuery = eventDao.queryBuilder();
String[] columns = {...};
eventsQuery.selectColumns(columns);
try {
List<EventModel> models;
models = eventsQuery.orderBy("timeStamp", false).query();
if (models == null || models.size() == 0) {
return;
}
ArrayList<PayloadSecureDataToolKit> toSendList = new ArrayList<>();
for (EventModel eventModel : models) {
try {
PayloadSecureDataToolKit payloadSecureDataToolKit = new PayloadSecureDataToolKit();
if (eventModel != null) {
// map my items ... not shown
toSendList.add(payloadSecureDataToolKit);
}
} catch (Exception e) {
Log.e(TAG, "Error adding payload! " + e + " ..... Skipping entry");
}
}
doAllNetworkCalls(toSendList);
} catch (SQLException e) {
e.printStackTrace();
}
}
my Retrofit stuff
public class ClientSecureDataToolKit {
private static ClientSecureDataToolKit mClientSecureDataToolKit;
private static Retrofit mRetrofit;
private ClientSecureDataToolKit(){
mRetrofit = new Retrofit.Builder()
.baseUrl(Utilities.getSecureDataToolkitURL())
.addCallAdapterFactory(RxJavaCallAdapterFactory.create())
.addConverterFactory(GsonConverterFactory.create())
.build();
}
public static ClientSecureDataToolKit getClientSecureDataKit(){
if(mClientSecureDataToolKit == null){
mClientSecureDataToolKit = new ClientSecureDataToolKit();
}
return mClientSecureDataToolKit;
}
public Observable<Record> putUserDataToSDK(PayloadSecureDataToolKit payloadSecureDataToolKit){
InterfaceSecureDataToolKit interfaceSecureDataToolKit = mRetrofit.create(InterfaceSecureDataToolKit.class);
Observable<Record> observable = interfaceSecureDataToolKit.putRecord(NetworkUtils.SECURE_DATA_TOOL_KIT_AUTH, payloadSecureDataToolKit);
return observable;
}
}
public interface InterfaceSecureDataToolKit {
#Headers({
"Content-Type: application/json"
})
#POST("/api/create")
Observable<Record> putRecord(#Query("api_token") String api_token, #Body PayloadSecureDataToolKit payloadSecureDataToolKit);
}
Update. I have been trying to apply this answer to not much luck. I am running out of steam for tonight. I am trying to implement this as a unit test, like I did for the original call for one item.. It looks like something is not right with use of lambda maybe..
public class RxJavaBatchTest {
Context context;
final static List<EventModel> models = new ArrayList<>();
#Before
public void before() throws Exception {
context = new MockContext();
EventModel eventModel = new EventModel();
//manually set all my eventmodel data here.. not shown
eventModel.setSampleId("SAMPLE0");
models.add(eventModel);
eventModel.setSampleId("SAMPLE1");
models.add(eventModel);
eventModel.setSampleId("SAMPLE3");
models.add(eventModel);
}
#Test
public void testSetupData() {
Assert.assertEquals(3, models.size());
}
#Test
public void testBatchSDK_Upload() {
Callable<List<EventModel> > callable = new Callable<List<EventModel> >() {
#Override
public List<EventModel> call() throws Exception {
return models;
}
};
Observable.fromCallable(callable)
.flatMapIterable(models -> models)
.flatMap(eventModel -> {
PayloadSecureDataToolKit payloadSecureDataToolKit = new PayloadSecureDataToolKit(eventModel);
return doNetworkCall(payloadSecureDataToolKit) // I assume this is just my normal network call.. I am getting incompatibility errors when I apply a testsubscriber...
.subscribeOn(Schedulers.io());
}, true, 1);
}
private Observable<Record> doNetworkCall(PayloadSecureDataToolKit payloadSecureDataToolKit) {
ClientSecureDataToolKit clientSecureDataToolKit = ClientSecureDataToolKit.getClientSecureDataKit();
Observable observable = clientSecureDataToolKit.putUserDataToSDK(payloadSecureDataToolKit);//.subscribe((Observer<? super Record>) testSubscriber);
return observable;
}
Result is..
An exception has occurred in the compiler (1.8.0_112-release). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
com.sun.tools.javac.code.Symbol$CompletionFailure: class file for java.lang.invoke.MethodType not found
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':app:compile<MyBuildFlavorhere>UnitTestJavaWithJavac'.
> Compilation failed; see the compiler error output for details.
Edit. No longer trying Lambdas. Even after setting up the path on my mac, javahome to point to 1.8, etc. I could not get it to work. If this was a newer project I would push harder. However as this is an inherited android application written by web developers trying android, it is just not a great option. Nor is it worth the time sink to get it working. Already into the days of this assignment instead of the half day it should have taken.
I could not find a good non lambda flatmap example. I tried it myself and it was getting messy.

If I understand you correctly, you want to make your calls in parallel?
So rx-y way of doing this would be something like:
Observable.fromCallable(() -> eventsQuery.orderBy("timeStamp", false).query())
.flatMapIterable(models -> models)
.flatMap(model -> {
// map your model
//avoid throwing exceptions in a chain, just return Observable.error(e) if you really need to
//try to wrap your methods that throw exceptions in an Observable via Observable.fromCallable()
return doNetworkCall(someParameter)
.subscribeOn(Schedulers.io());
}, true /*because you don't want to terminate a stream if error occurs*/, maxConcurrent /* specify number of concurrent calls, typically available processors + 1 */)
.subscribe(result -> {/* handle result */}, error -> {/* handle error */});
In your ClientSecureDataToolKit move this part into constructor
InterfaceSecureDataToolKit interfaceSecureDataToolKit = mRetrofit.create(InterfaceSecureDataToolKit.class);