I have an infinite observable (db observer) that emits finite sequences of items, that need to be processed and then emitted back as a group.
The problem here is, how to write it so that toList doesn't wait for the original source to finish, but for the flatMapIterable generated sequence instead;
DbObservable.map(new Func1<Query, List<Item>>() {
#Override
public List<Item> call(Query query) {
return query.items; //get items from db query
}
})
.flatMapIterable(new Func1<List<Item>, Iterable<Item>>() {
#Override
public Iterable<Item> call(List<GeoStop> geoStops) {
return geoStops;
}
})
.flatMap(/*process*/)
.toList() // regenerate List that was passed in to flatMapIterable
//subscribe and emit list of items
Nothing reaches subscribers due to toList being stuck waiting for DbObservable's onComplete.
toList() waits for onCompleted() event but the flatMap(/*process*/) doesn't propagate complete.
So, you need to call them inside a new flatMap()
db.map(q -> q.items)
.flatMap(items -> Observable.from(items)
.flatMapIterable(items)
.flatMap(/*process*/)
.toList()
)
.subscribe(...)
You could process the List inside the first flatMap with the flatMap+toList:
db.map(q -> q.items)
.flatMap(items -> Observable.from(items).flatMap(...).toList())
.subscribe(...)
Related
Let's say I have a function that takes a String and a long and returns a Single<String>.
Single<String> stringAddition(String someString, long value) {
return Single.just(someString + Long.toString(value));
}
Now I have this Observable...
Observable.interval(1, SECONDS)
.scan("", (cumulativeString, item) -> {
// Need to return the result of stringAddition(cummulativeString, item)
});
I'm at a loss on how to do this. Scan needs me to return a String, but II would like to use the method that returns a Single<String>. To me it seems like I need something that can combine the behaviour of both scan and flatMap. Is there any RxJava2 wizardry that can help me?
You can achieve it as follows. This can be somewhat shortened if stringAddition would have returned Observable
Observable<String> scanned = Observable.interval(1, TimeUnit.SECONDS)
.scan(
Observable.just(""),
(cumulativeString, item) ->
cumulativeString
.flatMap(str -> stringAddition(str, item).toObservable())
)
.flatMap(it -> it);
I have a Source as a cold Observable<List<T>> that emits elements in chunks (lists), I want to process every single item from the chunk in a separate thread, while the emiter (source) is waiting for termination of processing all items from the emited chunk to proceed with the next one and so on.
This code (rxjava 2.0.6) do the stuff but only in one Thread. If I want to fork the observer-computation in many Threads with observeOn(Schedulers.io), the source-thread continue emiting everything until completed and do not block by every chunk.
Observable<List<T>> lazy_source = Observable.create((ObservableEmitter<List<T>> e)
-> {
for (int i = 0; i < 1000; i++) {
List<T> chunk = produceChunkOf(10);
e.onNext(chunk);
}
e.onComplete();
});
lazy_source
.subscribeOn(Schedulers.io())
.flatMap(chunk ->
Observable.fromIterable(chunk)
// .observeOn(Schedulers.io()) // Uncommenting this will flat all 1000 chunks at once.
.doOnNext(item -> consume(item))
, 10) // Number of concurent Threads
.subscribe();
I will appreciate any help.
how about something like this:
Observable.range(0, 1000)
.concatMap(new Func1<Integer, Observable<?>>() {
#Override
public Observable<?> call(Integer integer) {
return produceChunkOf(10)
.flatMap(new Func1<Object, Observable<?>>() {
#Override
public Observable<?> call(Object item) {
return consume(item)
.observeOn(Schedulers.io());
}
}, 10)
.toList();
}
});
first, you create an Observable that emits the inputs to the produceChunkOf, then for each input item,you concatMap for your requirement for sequential execution for each chunk, for each input you produce the chunk , and process it in parallel with flatMap, then collect it after all items processed using toList()
here is the final version (without overhead):
Observable.range(0, 1_000_000)
.subscribeOn(Schedulers.io())
.concatMap(i -> produceChunkOf(100) // this returs an Observable of 100 items
.flatMap(item -> Observable
.just(item)
.observeOn(Schedulers.io())
.doOnNext(element -> consume(element)),
50)) // Number of concurent Threads
.subscribe();
1) I have method Observable> moviesWithoutGenres() which returns List of movies but movie's genres field is null
2) I have method Observable> movieGenres(Movie m) that returns list of genres for specified field
I need to implement method that should returns Observable> and each movie in the list will have list of genres.
I have implemented such stream but my solution is using transformation to Observable using Observable.from() and than to Observable> using toList() operator. This solution is not acceptable as I am using SQLBrite wrapper for reactive sql queries under the hood of first method. The onCompleted is not called as streams always remain open so that changes to the sql tables can be propagated to all subscribers. So, toList operator can't be performed.
public class Movie {
private String id;
private String title;
private List<String> genres;
...
}
1) Observable<List<Movie>> moviesWithoutGenres();
2) Observable<List<String>> movieGenres(Movie m);
Should be implemented:
Observable<List<Movie>> movies();
public Observable<List<Movie>> movies() {
return moviesWithoutGenres()
.switchMap(movies -> Observable.from(movies) // This is an Observable<Movie>
.concatMap(movie -> movieGenres(movie)
.first()
.map(genres -> new Movie(movie.id, movie.title, genres)))
.toList());
}
Observable.from() will emit the items in the list followed by an onComplete(). The trick is since movieGenres is an endless observable you have to .first() it so that onComplete() is called and toList() will work properly
class MyItem {
private param1, param2, param3;
}
MyItem item = new MyItem();
computeParam1(item);
computeParam2(item);
computeParam3(item);
waitForAllParamsToBeSet();
Each of the steps is independent from each other, and each step write the paramter into the object as final result.
The methods are completely different from their logic, no recursion.
How could I parallelize those steps, if possible at all?
Start Futures and then wait for results before assigning.
Future<Type1> item1 = ComputeParam1();
Future<Type2> item2 = ComputeParam2();
Future<Type3> item2 = ComputeParam3();
MyItem item = new MyItem();
assignParam1(item1.get());
assignParam2(item2.get());
assignParam3(item3.get());
As all computeParamX() accept one MyItem argument and have void return, they have a signature of Consumer<MyItem>. So you can parallelize their execution calling them in .forEach() of parallel stream, as follows:
final MyItem item = new MyItem();
Stream.<Consumer<MyItem>>of(this::computeParam1, this::computeParam2, this::computeParam3)
.parallel()
.forEach(c -> c.accept(item));
As .forEach() is terminal operation, it will block until all operations complete, so you can safely use item object after it returns.
In Java 8 you could simply create your collection of tasks as next:
Collection<Runnable> tasks = Arrays.asList(
() -> System.out.println("Compute param1"),
() -> System.out.println("Compute param2"),
() -> System.out.println("Compute param3")
);
Then launch the tasks in parallel
tasks.parallelStream().forEach(Runnable::run);
I want to perform a stream where the output from the stream is then used as the source for the same stream, in the same operation.
I currently perform this sort of operation using a queue; I remove an item, process it, and add any results that need further processing back to the queue. Here are two examples of this sort of thing:
Queue<WorkItem> workQueue = new Queue<>(workToDo);
while(!workQueue.isEmpty()){
WorkItem item = workQueue.remove();
item.doOneWorkUnit();
if(!item.isDone()) workQueue.add(item);
}
Queue<Node> nodes = new Queue<>(rootNodes);
while(!nodesLeft.isEmpty()){
Node node = nodes.remove();
process(node);
nodes.addAll(node.children());
}
I would imagine that the first could be performed concurrently like this:
try {
LinkedBlockingQueue<WorkItem> workQueue = new LinkedBlockingQueue<>();
Stream<WorkItem> reprocess = Stream.generate(() -> workQueue.remove()).parallel();
Stream.concat(workToDo.parallelstream(), reprocess)
.filter(item -> {item.doOneWorkUnit(); return !item.isDone();})
.collect(Collectors.toCollection(() -> workQueue));
} catch (NoSuchElementException e){}
And the second as:
try {
LinkedBlockingQueue<Node> reprocessQueue = new LinkedBlockingQueue<>();
Stream<WorkItem> reprocess = Stream.generate(() -> nodes.remove()).parallel();
Stream.concat(rootNodes.parallelStream(), reprocess)
.filter(item -> {process(item); return true;})
.flatMap(node -> node.children().parallelStream())
.collect(Collectors.toCollection(() -> reprocessQueue));
} catch (NoSuchElementException e){}
However, these feel like kludgy workarounds, and I dislike having to resort to using exceptions. Does anyone have a better way to do this sort of thing?
To make work parallel, I would use standard java.util.concurrent.Executor. To return the task back to working queue, in the end of the code of each task, add executor.execute(this).