I´m playing with Spring reactor, and I cannot see any differences between concat and merge operator
Here's my example
#Test
public void merge() {
Flux<String> flux1 = Flux.just("hello").doOnNext(value -> {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
});
Flux<String> flux2 = Flux.just("reactive").doOnNext(value -> {
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
e.printStackTrace();
}
});
Flux<String> flux3 = Flux.just("world");
Flux.merge(flux1, flux2, flux3)
.map(String::toUpperCase)
.subscribe(System.out::println);
}
#Test
public void concat() {
Flux<String> flux1 = Flux.just("hello").doOnNext(value -> {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
});
Flux<String> flux2 = Flux.just("reactive").doOnNext(value -> {
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
e.printStackTrace();
}
});
Flux<String> flux3 = Flux.just("world");
Flux.concat(flux1, flux2, flux3)
.map(String::toUpperCase)
.subscribe(System.out::println);
}
Both behave exactly the same. Can someone explain the difference between the two operations?
The essential difference between merge and concat is that in merge, both streams are live. In case of concat, first stream is terminated and then the other stream is concatenated to it.
Concat
Merge
The difference is already mentioned in the API docs that while concat first reads one flux completely and then appends the second flux to that, merge operator doesn't guarantee the sequence between the two flux.
In order to see the difference, modify your merge() code as below.
e.g. sample code below
//Flux with Delay
#Test
public void merge() {
Flux<String> flux1 = Flux.just("Hello", "Vikram");
flux1 = Flux.interval(Duration.ofMillis(3000))
.zipWith(flux1, (i, msg) -> msg);
Flux<String> flux2 = Flux.just("reactive");
flux2 = Flux.interval(Duration.ofMillis(2000))
.zipWith(flux2, (i, msg) -> msg);
Flux<String> flux3 = Flux.just("world");
Flux.merge(flux1, flux2, flux3)
.subscribe(System.out::println);
try {
Thread.sleep(8000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
As you modify the Flux.interval duration, currently set as 3000 milliseconds you will see that the output with merge() keeps changing. But with concat(), the output will be always same.
Another noteworthy difference is that all variations of concat subscribe to the second stream lazily, only after the first stream has terminated.
Whereas all variations of merge subscribe to the publishers eagerly (all publishers are subscribed together)
Running the below code highlights this aspect:
//Lazy subscription of conact
Flux.concat(Flux.just(1, 2, 3, 4).delayElements(Duration.ofMillis(500)),
Flux.just(10, 20, 30, 40).delayElements(Duration.ofMillis(500)))
.subscribe(System.out::println, System.out::println);
//Eager subscription of the merge. Also, try mergeSequential.
Flux.merge(Flux.range(500, 3).delayElements(Duration.ofMillis(500)),
Flux.range(-500, 3).delayElements(Duration.ofMillis(300)))
.subscribe(System.out::println, System.out::println);
Related
I have following proactive method.
void acceptSome(Consumer<? super String> consumer) {
consumer.accept("A");
consumer.accept("B");
}
And I wrote following method for reactor.
void emitAll(Sinks.Many<String> sink) {
try {
acceptSome(e -> {
sink.emitNext...(e);
});
sink.emitComplete...
} catch (Exception e) {
sink.emitError...
}
}
And I tested with following two methods, one with full buffer and the other with single buffer using thread.
#Test
void emitAll__onBackpressureBufferAll() {
final var sink = Sinks.many().unicast().<String>onBackpressureBuffer();
TheClass.emitAll(sink);
sink.asFlux()
.doOnNext(e -> {
log.debug("next: {}", e);
})
.blockLast();
}
#Test
void emitAll__onBackpressureBufferOne() {
var sink = Sinks.many().unicast().<String>onBackpressureBuffer(new ArrayBlockingQueue<>(1));
new Thread(() -> {
sink.asFlux()
.doOnNext(e -> {
log.debug("next: {}", e);
})
.blockLast();
}).start();
TheClass.emitAll(sink);
}
Now, How can I (Can I do that?) implement a method accepts a sink and returns a Flux or CompletableFuture<Flux> so that caller simply subscribe to the result without thread, with minimum buffer?
CompletableFuture<Flux<String>> emitAllAsync(Sinks.Many<String> sink) {
}
Thank you.
I tried some and it works yet doesn't seem righteous.
void emitAll(Sinks.Many<String> sink, Semaphore semaphore) {
try {
acceptSome(v -> {
try {
semaphore.acquire();
} catch (final InterruptedException ie) {
Thread.currentThread().interrupt();
throw new RuntimeException(ie);
}
log.debug("emitting {}", v);
sink.tryEmitNext(v).orThrow();
});
sink.tryEmitComplete().orThrow();
} catch (final IOException ioe) {
log.error("failed to emit to {}", sink, ioe);
sink.tryEmitError(ioe).orThrow();
}
}
CompletableFuture<Flux<String>> emitAllAsync() {
var sink = Sinks.many().unicast().<String>onBackpressureBuffer(new ArrayBlockingQueue<>(1));
var semaphore = new Semaphore(1);
CompletableFuture
.runAsync(() -> emitAll(sink, semaphore));
return CompletableFuture.completedFuture(sink.asFlux().doOnNext(v -> semaphore.release()));
}
I have 2 CompletableFutures. The task2 should only start once task1 finishes. Then, I need to wait for all tasks to finish. In my code below, the program ends after task1 ends. The task2 starts but did not finish. Any ideas why this happens? Also, why is it that the list only contains 1 entry while in the code, I added 2?
Code:
public void testFutures () throws Exception {
List<CompletableFuture<Void>> futures = new ArrayList<>();
CompletableFuture<Void> task1 = CompletableFuture.supplyAsync( () -> {
System.out.println(" task1 start");
try {
Thread.sleep(5000L);
} catch (InterruptedException ex) {
ex.printStackTrace();
}
System.out.println(" task1 done");
return null;
});
task1.whenComplete( (x, y) -> {
CompletableFuture<Void> task2 = CompletableFuture.supplyAsync( () -> {
System.out.println(" task2 start");
try {
Thread.sleep(2000L);
} catch (InterruptedException ex) {
ex.printStackTrace();
}
System.out.println(" task2 done");
return null;
});
futures.add(task2);
});
futures.add(task1);
// wait for the calls to finish
try {
CompletableFuture.allOf(futures.toArray(new CompletableFuture[futures.size()])).whenComplete( (x, y) -> {
System.out.println(" all tasks done " + futures.size());
}).get();
} catch (Exception e) {
e.printStackTrace();
}
}
Output:
task1 start
task1 done
all tasks done 1
task2 start
Let's first clean your code.
Let's define a method that will do the sleeping, so that it does not muddy the water:
private static void sleep(int seconds) {
try {
Thread.sleep(TimeUnit.SECONDS.toMillis(seconds));
} catch (InterruptedException ex) {
throw new RuntimeException(ex);
}
}
Then let's separate the tasks and use proper methods:
private static CompletableFuture<Void> task1() {
return CompletableFuture.runAsync(() -> {
System.out.println(" task1 start");
sleep(5);
System.out.println(" task1 done");
});
}
private static CompletableFuture<Void> task2() {
return CompletableFuture.runAsync(() -> {
System.out.println(" task2 start");
sleep(2);
System.out.println(" task2 done");
});
}
You need to understand that chaining of CompletableFuture methods already do exactly what you want, they run the next stage, after the previous one has ended. You can make your code far, far more easy with:
public static void main(String[] args) throws Exception {
testFutures();
}
private static void testFutures() throws Exception {
CompletableFuture<Void> both = task1().thenCompose(ignoreMe -> task2());
both.get();
System.out.println("both done");
}
You have two problems.
First, you've created a race condition as to when task2 gets added to your list of futures. At the time you execute this line—
CompletableFuture.allOf(...).get();
—which I'll call the terminating getter, you only have task1 in the list. See for yourself by outputting its size:
// wait for the calls to finish
try {
System.out.println("# of futures: " + futures.size()); // 1
task2 still runs eventually, because you scheduled it with whenComplete(). But it's not your terminating getter that triggers it.
Recall that I said it's a race condition. To demonstrate this for yourself, add a sleep() before the terminating getter, like so:
try {
Thread.sleep(6000L);
} catch (InterruptedException ex) {
ex.printStackTrace();
}
// wait for the calls to finish
try {
System.out.println("# of futures: " + futures.size()); // 2
Then you'll have given it enough time to add task2.
But here's the thing. Now is the terminating getter triggering both tasks?
Still no! And that's the second problem: You almost always want to use one of the thenRun(), thenAccept(), thenApply(), thenCompose() methods. These methods chain your futures, i.e. make each stage dependent on the previous, so that your terminating getter actually waits for the entire chain to complete. whenComplete() is a special method that kicks off a totally unrelated pipeline and is thus unaffected by the terminating get().
In your case, you want to use thenRun(), like this:
task1.thenRun( ignore -> {
Okay, so how do we combine all that?
public static void testFutures () throws Exception {
CompletableFuture<Void> task1 = CompletableFuture.supplyAsync( () -> {
System.out.println(" task1 start");
try {
Thread.sleep(5000L);
} catch (InterruptedException ex) {
ex.printStackTrace();
}
System.out.println(" task1 done");
return null;
});
CompletableFuture<Void> futuresChain = task1.thenRun( () -> {
System.out.println(" task2 start");
try {
Thread.sleep(2000L);
} catch (InterruptedException ex) {
ex.printStackTrace();
}
System.out.println(" task2 done");
});
// wait for the calls to finish
try {
futuresChain.thenRun( () -> {
System.out.println(" all tasks done ");
}).toCompletableFuture().get();
} catch (Exception e) {
e.printStackTrace();
}
}
Output:
task1 start
task1 done
task2 start
task2 done
all tasks done
You see, you only need to supplyAsync() for the first task. You want to run task2 sequentially after that task, so thenRun() will do the scheduling (the supplyAsync()ing) for you. So you don't need an array of futures either. The allOf() is for when you want to run tasks in parallel, and wait for all of them to finish.
I am trying to iterate over an array of Maps and do some asynchronous actions. I have tried a few things using the RxJava library, but everything I've tried seems to be synchronous. I am trying to avoid creating new threads manually and want to let RxJava handle it. This is what I've tried so far.
Observable.from(new Map[20])
.subscribeOn(Schedulers.newThread())
.observeOn(Schedulers.computation())
.forEach(batch -> {
try {
System.out.println(1);
Thread.sleep(3000);
System.out.println(2);
} catch (Exception e) {
}
});
Observable.from(new Map[20])
.subscribeOn(Schedulers.newThread())
.observeOn(Schedulers.computation())
.subscribe(batch -> {
try {
System.out.println(1);
Thread.sleep(3000);
System.out.println(2);
} catch (Exception e) {
}
});
Observable.from(new Map[20])
.subscribeOn(Schedulers.newThread())
.subscribe(batch -> {
try {
System.out.println(1);
Thread.sleep(3000);
System.out.println(2);
} catch (Exception e) {
}
});
Observable.from(new Map[20])
.subscribe(batch -> {
try {
System.out.println(1);
Thread.sleep(3000);
System.out.println(2);
} catch (Exception e) {
}
});
When I run unit tests with the code above I see the following output.
1
2
1
2
1
2
...
What I want to see is
1
1
1
...
2
2
2
How do I iterate asynchronously over a Map array using RxJava?
You can achieve it changing from Observable to Flowable and use parallel:
Flowable.fromIterable(array)
.parallel(3) // number of items in parallel
.runOn(Schedulers.newThread()) // the desired scheduler
.map(item -> {
try {
System.out.println(1);
Thread.sleep(3000);
System.out.println(2);
} catch (Exception e) {
}
return Completable.complete();
})
.sequential().subscribe();
If you're stuck using RxJava 1.x then you wont have access to the Flowable class. This wasn't my case, but something like the below code can do parallel actions. There is more nesting, but it works.
final ExecutorService executor = Executors.newFixedThreadPool(2);
List<String> iterableList = new ArrayList<>();
iterableList.add("one");
iterableList.add("two");
iterableList.add("three");
iterableList.add("4");
iterableList.add("5");
iterableList.add("6");
iterableList.add("7");
iterableList.add("8");
iterableList.add("9");
Observable.from(iterableList)
.flatMap(val -> Observable.just(val)
.subscribeOn(Schedulers.from(executor))
.doOnNext(numString -> {
try {
System.out.println(1);
Thread.sleep(500);
System.out.println(2);
} catch (Exception ex) {
}
})
)
.subscribe();
I have this code where I execute sets of callables, I need one set to finish all it's work before triggering the next set. This code seems to work fine but sometimes next set would start running before time. What is wrong here?
private void executeSubGraph(QuestExecutionContext ctx, Set<Activity> subGraph, int progressAfterRan) {
ExecutorService pool = Executors.newFixedThreadPool(16);
subGraph.forEach(a -> {
ActivityRunner<? extends Activity> runner = activityRunnerFactory.getRunner(ctx, a);
if (runner != null) {
Callable<List<PortValuePart>> runnerCallable = () -> {
try {
LOG.info("Running {} in {}", a, a.getClass() );
List<PortValuePart> result = runner.call();
LOG.info("Result of {} in {} is {}", a, a.getClass(), result);
if (result != null) {
result.forEach(r -> resultProcessor.processResult(new PortValuePartEnvelope(r)));
}
return result;
} catch (Exception e) {
LOG.warn("Exception for {} in {}", a, runner.getClass(), e);
resultProcessor.processResult(Progress.failed(ctx.getId(), e));
throw new RuntimeException(e);
}
};
Future<List<PortValuePart>> p = pool.submit(runnerCallable);
} else {
LOG.warn("No runner found for activity {}", a);
resultProcessor.processResult(Progress.failed(ctx.getId(), new RuntimeException("No runner found for activity " + a)));
throw new RuntimeException("No runner found for activity " + a);
}
});
pool.shutdown();
try {
pool.awaitTermination(WAIT_TIME_MILLIS, TimeUnit.MILLISECONDS);
resultProcessor.processResult(Progress.running(ctx.getId(), progressAfterRan));
} catch (InterruptedException e) {
throw new PlatformException("Execution interrupted.");
}
}
Note that ExecutorService.awaitTermination doesn't throw an exception if it times out; it just returns false. If you want to make sure that the next calls don't run concurrently with these ones, you should probably use the return value, and maybe throw an exception (and kill the tasks) if it's taking way too long.
I'm trying to convert this RxJava1 code to RxJava2
public static Observable<Path> listFolder(Path dir, String glob) {
return Observable.<Path>create(subscriber -> {
try {
DirectoryStream<Path> stream =
Files.newDirectoryStream(dir, glob);
subscriber.add(Subscriptions.create(() -> {
try {
stream.close();
} catch (IOException e) {
e.printStackTrace();
}
}));
Observable.<Path>from(stream).subscribe(subscriber);
} catch (DirectoryIteratorException ex) {
subscriber.onError(ex);
} catch (IOException ioe) {
subscriber.onError(ioe);
}
});
}
The thing is that in Rxjava2 I don't get a subscriber to add a new subscription to it.
Enjoy RxJava 2 conciseness (Flowable is the backpressure supporting class now):
public static Flowable<Path> listFolder(Path dir, String glob) {
return Flowable.using(
() -> Files.newDirectoryStream(dir, glob),
stream -> Flowable.fromIterable(stream),
stream -> stream.close());
}
If you don't want backpressure then replace Flowable with Observable.