I'm new with RxJava. I'm trying to create an observable that when it completes it will start all over again until I call dispose, but I'm facing an OutofMemory error after a while, below is a simplified example of what I'm trying to do
public void start() throws RuntimeException {
log.info("\t * Starting {} Managed Service...", getClass().getSimpleName());
try {
executeObserve();
log.info("\t * Starting {} Managed Service...OK!", getClass().getSimpleName());
} catch (Exception e) {
log.info("Managed Service {} FAILED! Reason is {} ", getClass().getSimpleName(), e.getMessage(), e);
}
}
start is invoked at the initialization phase once, the executeObserve is as follows (in a simplified form..). Notice that on the onComplete I "resubmit" executeObserve
public void executeObserve() throws RuntimeException {
Observable<Book> booksObserve
= manager.getAsObservable();
booksObserve
.map(Book::getAllOrders)
.flatMap(Observable::fromIterable)
.toList()
.subscribeOn(Schedulers.io())
.subscribe(collectedISBN ->
Observable.fromIterable(collectedISBN)
.buffer(10)
// ...some more steps here...
.toList()
.toObservable()
// resubmit
.doOnComplete(this::executeObserve)
.subscribe(validISBN -> {
// do something with the valid ones
})
)
);
}
My guess is that this is not the way to go if I want to resubmit my tasks but it was not possible to find any documentation.
the booksObserve is implemented as follows
public Observable<Book> getAsObservable() {
return Observable.create(e -> {
try (CloseableResultSet<Book> rs = (CloseableResultSet<Book>) datasource.retrieveAll())) {
for (Book r : rs) {
e.onNext(r);
}
e.onComplete();
} catch (Exception ex) {
e.onError(ex);
}
});
}
What is the correct way to constantly resubmit an operation until we call dispose or equivalent? I'm using RxJava 2
You have created an endless recursion, the loop will create more and more resources and sometime it will blow with OutOfMemory/Stack overflow exception.
In order to repeat the Observable work you should use repeat() operator, it will resubscribes to the Observable when it receives onComplete().
Besides that, some general comments on your code:
why are you nesting the second Observable inside the subscriber? you are breaking the chain, you can just continue the chain instead of creating new Observable at the Subscriber.
Moreover, it's seems (assuming Observable.fromIterable(collectedBets) using the collectedISBN that gets with the onNext() o.w. from where does it comes?) you're collecting all items to a list, and then flatting it again using from iterable, so it's seems you can just continue on the stream , something like that:
booksObserve
.map(Book::getAllOrders)
.flatMap(Observable::fromIterable)
.buffer(10)
// ...some more steps here...
.toList()
.toObservable()
// resubmit
.doOnComplete(this::executeObserve)
.subscribeOn(Schedulers.io())
.subscribe(validISBN -> {
// do something with the valid ones
});
Anyhow, with the nested Observable, the repeat() operator will just repeat the nested one, and not the entire stream (which is what you want) as it is not connected to it.
In continuation to my question the repeat as #yosriz suggested is the proper way to go, the following simple snippet demonstrates that the observable source will be called on each repeat
Observable<Integer> recursiveObservable = Observable.create(emitter -> {
System.out.println("Calling to emit data");
Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 0).forEach(emitter::onNext);
emitter.onComplete();
});
recursiveObservable
.buffer(2)
.repeat()
.subscribe(integers -> {
System.out.println(integers);
TimeUnit.SECONDS.sleep(1);
});
Related
I have an Observable that at some point has to write things to the cache - and we would like to wait that writes are done before finishing the whole operation on the observable (for reporting purposes).
For the purpose of test, the cache write Completable looks like this:
Completable.create(
emitter ->
new Thread(
() -> {
try {
Thread.sleep(2000);
doSomething();
emitter.onComplete();
} catch (InterruptedException e) {
e.printStackTrace();
}
})
.start());
Since I have several cache writes, I try to merge them in a container class:
public class CacheInsertionResultsTracker {
private Completable cacheInsertResultsCompletable;
public CacheInsertionResultsTracker() {
this.cacheInsertResultsCompletable = Completable.complete();
}
public synchronized void add(Completable cacheInsertResult) {
this.cacheInsertResultsCompletable = this.cacheInsertResultsCompletable.mergeWith(cacheInsertResult);
}
public Completable getCompletable() {
return this.cacheInsertResultsCompletable;
}
}
And I try to merge it with Observable in a following way:
CacheInsertionResultsTracker tracker = new ...;
observable
.doOnNext(next->tracker.add(next.writeToCache(...)))
.mergeWith(Completable.defer(()->tracker.getCompletable()))
.subscribe(
// on next
this::logNextElement
// on error
this::finishWithError
// on complete
this::finishWithSuccess
);
How could I make sure that by the time finishWithSuccess is called the doSomething is completed?
The problem is that the Completable reference is updated every time I add a new one, and it happens after the mergeWith runs...
The solution that seems to work for our use case is to use concatWith + defer:
observable
.doOnNext(next->tracker.add(next.writeToCache(...)))
.concatWith(Completable.defer(()->tracker.getCompletable()))
.subscribe(
// on next
this::logNextElement
// on error
this::finishWithError
// on complete
this::finishWithSuccess
);
Concat assures that the subscription to the Completable happens only after the Observable is done, and defer defers getting the final Completable till this subscription (so all the objects are already added to the tracker).
Based on the comments, you could replace the completable cache with ReplaySubject<Completable>, do some timeout to detect inactivity and have the observable sequence end.
ReplaySubject<Completable> cache = ReplaySubject.create();
cache.onNext(completable);
observable.mergeWith(
cache.flatMapCompletable(v -> v)
.timeout(10, TimeUnit.MILLISECONDS, Completable.complete())
)
Edit:
Your updated example implies you want to run Completables in response to items in the main observable, isolated to that sequence, and wait for all of them to complete. This is a typical use case for flatMap:
observable.flatMap(
next -> next.writeToCache(...).andThen(Observable.just(next))
)
.subscribe(
this::logNextElement
// on error
this::finishWithError
// on complete
this::finishWithSuccess
);
The problem with the following code is that the subscriber sees only items of the first flux (i.e. only printing 1). Interestingly, if I add delayElements, it works fine.
This is a toy example, but my intention is to replace it with Flux's that make HTTP GET requests and emit their results (also, could be more than two).
So reformulating my question, I have a many-to-one relation that needs to be implemented. How to implement it, considering my case? Would you use some kind of Processor?
public static void main(String[] args) throws Exception {
Flux<Integer> flux1 = Flux.generate(emitter -> {
emitter.next(1);
});
Flux<Integer> flux2 = Flux.generate(emitter -> {
emitter.next(2);
});
Flux<Integer> merged = flux1.mergeWith(flux2);
merged.subscribe(s -> System.out.println(s));
Thread.currentThread().join();
}
Trying to achieve the same idea with a TopicProcessor but it suffers from the same issue:
public static void main(String[] args) throws Exception {
Flux<Integer> flux1 = Flux.generate(emitter -> {
emitter.next(1);
try {
Thread.sleep(100);
} catch (Exception e) {}
});
Flux<Integer> flux2 = Flux.generate(emitter -> {
emitter.next(2);
try {
Thread.sleep(100);
} catch (Exception e) {}
});
TopicProcessor<Integer> processor = TopicProcessor.create();
flux1.subscribe(processor);
flux2.subscribe(processor);
processor.subscribe(s -> System.out.println(s));
Thread.currentThread().join();
}
From the docs:
Note that merge is tailored to work with asynchronous sources or finite sources. When dealing with an infinite source that doesn't already publish on a dedicated Scheduler, you must isolate that source in its own Scheduler, as merge would otherwise attempt to drain it before subscribing to another source.
You're creating an infinite source here without a dedicated scheduler, so it's attempting to drain that source fully before merging - and that's why you have your issue.
This may not be an issue in your real-world use case, since the result of the GET request, presumably, won't be infinite. However, if you want to make sure the results are interleaved regardless, you just need to make sure you set up each flux with its own scheduler (by calling subscribeOn(Schedulers.elastic()); on each Flux.)
So your example then becomes:
Flux<Integer> flux1 = Flux.<Integer>generate(emitter -> emitter.next(1))
.subscribeOn(Schedulers.elastic());
Flux<Integer> flux2 = Flux.<Integer>generate(emitter -> emitter.next(2))
.subscribeOn(Schedulers.elastic());
Flux<Integer> merged = flux1.mergeWith(flux2);
merged.subscribe(s -> System.out.println(s));
Thread.currentThread().join();
Is there any method like doOnNext, but async?
For example, I need to do some long logging (sent notification by email) for determined element.
Scheduler myParallel = Schedulers.newParallel("my-parallel", 4);
Flux<Integer> ints = Flux.just(1, 2, 3, 4, 5)
.publishOn(myParallel)
.doOnNext(v -> {
// For example, we need to do something time-consuming only for 3
if (v.equals(3)) {
try {
Thread.sleep(3000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
System.out.println("LOG FOR " + v);
});
ints.subscribe(System.out::println);
But why should I wait for logging of 3? I want to do this logic asynchronously.
Now I have only this solution
Thread.sleep(10000);
Scheduler myParallel = Schedulers.newParallel("my-parallel", 4);
Scheduler myParallel2 = Schedulers.newParallel("my-parallel2", 4);
Flux<Integer> ints = Flux.just(1, 2, 3, 4, 5)
.publishOn(myParallel)
.doOnNext(v -> {
Mono.just(v).publishOn(myParallel2).subscribe(value -> {
if (value.equals(3)) {
try {
Thread.sleep(3000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
System.out.println("LOG FOR " + value);
});
});
ints.subscribe(System.out::println);
Is there any "nice" solution for this?
If you're absolutely sure you don't care wether or not the email sending succeeds, then you could use "subscribe-inside-doOnNext" but I'm pretty confident that would be a mistake.
In order to have your Flux propagate an onError signal if the "logging" fails, the recommended approach is to use flatMap.
The good news is that since flatMap merges results from the inner publishers immediately into the main sequence, you can get still emit each element immediately AND trigger the email. The only caveat is that the whole thing will only complete once the email-sending Mono has also completed. You can also check within the flatMap lambda if the logging needs to happen at all (rather than inside the inner Mono):
//assuming sendEmail returns a Mono<Void>, takes care of offsetting any blocking send onto another Scheduler
source //we assume elements are also publishOn as relevant in `source`
.flatMap(v -> {
//if we can decide right away wether or not to send email, better do it here
if (shouldSendEmailFor(v)) {
//we want to immediately re-emit the value, then trigger email and wait for it to complete
return Mono.just(v)
.concatWith(
//since Mono<Void> never emits onNext, it is ok to cast it to V
//which makes it compatible with concat, keeping the whole thing a Flux<V>
sendEmail(v).cast(V.class)
);
} else {
return Mono.just(v);
}
});
Flux<Integer> ints = Flux.just(1, 2, 3, 4, 5)
.flatMap(integer -> {
if (integer != 3) {
return Mono.just(integer)
.map(integer1 -> {
System.out.println(integer1);
return integer;
})
.subscribeOn(Schedulers.parallel());
} else {
return Mono.just(integer)
.delayElement(Duration.ofSeconds(3))
.map(integer1 -> {
System.out.println(integer1);
return integer;
})
.subscribeOn(Schedulers.parallel());
}
}
);
ints.subscribe();
Following is my Current Code
private final List<Disposable> subscriptions = new ArrayList<>();
for (Instrument instrument : instruments) {
// Waiting for OrderBook to generate Reliable results.
GenericBook Book =
service
.getBook(instrument.getData())
.filter(gob -> onBookUpdate(gob))
.blockingFirst();
subscriptions.add(
service
.getBook(instrument.getData())
.subscribe(
gob -> {
try {
onBookUpdate(gob);
} catch (Exception e) {
logger.error("Error on subscription:", e);
}
},
e -> logger.error("Error on subscription:", e)));
}
So what it does is for each instrument it first Block wait till the output of onBookUpdate(gob) Becomes true. onBookUpdate(gob) returns boolean.
Once we have first onBookUpdate as true then i Will push that subscriber into subscriptions variable.
This slow down as I have to wait foreach instrument and then move on the next instrument.
My Goal is to run all these in parallel then wait all to finish and push them to subscriptions variable.
I tried zip but didn't work
List<Observable<GenericOrderBook>> obsList = null;
for (Instrument instrument : instruments) {
// This throws nullException.
obsList.add(service
.getBook(instrument.getData())
.filter(gob -> onBookUpdate(gob))
.take(1));
}
}
// Some how wait over here until all get first onBookUpdate as true.
String o = Observable.zip(obsList, (i) -> i[0]).blockingLast();
When using observables etc, one should embrace them wholeheartedly. One of the premises for embracing is to separate the configuration and construction of your pipeline from its execution.
In other words, configure your pipeline upfront and then, when the data is available, send the data through it.
Furthermore, embracing observables implies avoiding for-loops.
I'm not 100% what your use case is but what I'd suggest is to create a pipeline that takes an instrument as input and returns a subscription...
So something like
service.getBook(instrument.getData())
.flatMap(gob -> {
onBookUpdate(gob);
return gob;
});
That will return an Observable that you can subscribe to and add the result to the subscriptions.
Then create a seed observable that pumps the instrument objects into it.
Not sure of some of the details of your API, so come back to me if this is not clear or I've made a wrong assumption.
I am assuming instruments to be a List. If yes, then you can do something like this,
Observable
.fromIterable(instruments)
// Returns item from instrument list one by one and passes it to getBook()
.flatmap(
instrument -> getBook(instrument.getData())
)
.filter(
gob -> onBookUpdate(gob)
)
// onComplete will be called if no items from filter
.switchIfEmpty(Observable.empty())
.subscribe(
onBookUpdateResponse -> // Do what you want,
error -> new Throwable(error)
);
Hope this helps.
I'm consuming an API that returns CompletableFutures for querying devices (similar to digitalpetri modbus).
I need to call this API with a couple of options to query a device and figure out what it is - this is basically trial and error until it succeeds. These are embedded device protocols that I cannot change, but you can think of the process as working similar to the following:
Are you an apple?
If not, then are you a pineapple?
If not, then are you a pen?
...
While the API uses futures, in reality, the communications are serial (going over the same physical piece of wire), so they will never be executed synchronously. Once I know what it is, I want to be able to stop trying and let the caller know what it is.
I already know that I can get the result of only one of the futures with any (see below), but that may result in additional attempts that should be avoided.
Is there a pattern for chaining futures where you stop once one of them succeeds?
Similar, but is wasteful of very limited resources.
List<CompletableFuture<String>> futures = Arrays.asList(
CompletableFuture.supplyAsync(() -> "attempt 1"),
CompletableFuture.supplyAsync(() -> "attempt 2"),
CompletableFuture.supplyAsync(() -> "attempt 3"));
CompletableFuture<String>[] futuresArray = (CompletableFuture<String>[]) futures.toArray();
CompletableFuture<Object> c = CompletableFuture.anyOf(futuresArray);
Suppose that you have a method that is "pseudo-asynchronous" as you describe, i.e. it has an asynchronous API but requires some locking to perform:
private final static Object lock = new Object();
private static CompletableFuture<Boolean> pseudoAsyncCall(int input) {
return CompletableFuture.supplyAsync(() -> {
synchronized (lock) {
System.out.println("Executing for " + input);
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
return input > 3;
}
});
}
And a List<Integer> of inputs that you want to check against this method, you can check each of them in sequence with recursive composition:
public static CompletableFuture<Integer> findMatch(List<Integer> inputs) {
return findMatch(inputs, 0);
}
private static CompletableFuture<Integer> findMatch(List<Integer> inputs, int startIndex) {
if (startIndex >= inputs.size()) {
// no match found -- an exception could be thrown here if preferred
return CompletableFuture.completedFuture(null);
}
return pseudoAsyncCall(inputs.get(startIndex))
.thenCompose(result -> {
if (result) {
return CompletableFuture.completedFuture(inputs.get(startIndex));
} else {
return findMatch(inputs, startIndex + 1);
}
});
}
This would be used like this:
public static void main(String[] args) {
List<Integer> inputs = Arrays.asList(0, 1, 2, 3, 4, 5);
CompletableFuture<Integer> matching = findMatch(inputs);
System.out.println("Found match: " + matching.join());
}
Output:
Executing for 0
Executing for 1
Executing for 2
Executing for 3
Executing for 4
Found match: 4
As you can see, it is not called for input 5, while your API (findMatch()) remains asynchronous.
I think the best you can do is, after your retrieval of the result,
futures.forEach(f -> f.cancel(true));
This will not affect the one having produced the result, and tries its best to stop the others. Since IIUC you get them from an outside source, there's no guarantee it will actually interrupt their work.
However, since
this class has no direct control over the computation that causes it to be completed, cancellation is treated as just another form of exceptional completion
(from CompletableFuture doc), I doubt it will do what you actually want.