I have a function that return the input as a Mono:
public static Mono<Integer> emitter(int param){
return Mono.just(param)
.delayElement(Duration.ofMillis(100)); //delay to simulate http response
}
I'd like to call the emitter once with an initial value of 3 but then have it repeat until certain size is reached. This repeat logic should be in the main method so I can't modify the emitter().
public static void main(String[] args){
int maxSize = 5;
int initial = 3;
Mono<Integer> response = emitter(initial);
response
.doOnNext(s -> {
System.out.println("need more!");
})
.subscribe();
}
One naive solution would be:
public static void main(String[] args){
int maxSize = 5;
int initial = 3;
for(int i = 0; i < 999; i++) {
Mono<Integer> response = emitter(initial+i);
Mono<Boolean> isDone = response
.flatMap(elem -> {
if(elem < maxSize) {
System.out.println("need more!");
return Mono.just(false);
} else {
System.out.println("ok done!");
return Mono.just(true);
}
});
if(isDone.block())
break;
}
}
Basically, I'm trying to create another Mono with dynamic parameters based on the result of the previous Mono. I know that Mono/Flux are immutable...
Is there a neat and reactive way of doing this?
I've tried things like Flux.range(0, Integer.MAX_VALUE).zipWith(myMono) to try and feed parameters into the emitter but couldn't make it work.
PS. I know my example doesn't make much sense. I've tried to simplify my real world scenario that involves lists and spring WebFlux(emitter).
Thanks!
---EDIT
Ok, here's what I came up with:
public static void main(String[] args) throws InterruptedException {
int maxSize = 5;
int initial = 3;
Flux.range(initial, 10)
.delayElements(Duration.ofSeconds(1))
.flatMap(param -> emitter(param))
.flatMap(it -> {
if(it < maxSize) {
System.out.println("need more!: " + it);
return Mono.just(false);
} else {
System.out.println("done!: " + it);
return Mono.just(true);
}
})
.takeUntil(Boolean::booleanValue)
.subscribe();
Thread.sleep(6000);
}
need more!: 3
need more!: 4
done!: 5
One problem is that if I don't delay Flux.range, the execution is not done in order and it's possible for more or less print statements to be output than the 3 lines expected.
You can use the expand function of Publisher, which acts like recursion
e.g.
emitter(initial)
.expand(i -> i < maxSize ? emitter(i + 1) : Mono.empty())
.doOnNext(i -> System.out.println("i = " + i))
.subscribe();
Related
I have this code, where I have my own homemade array class, that I want to use to test the speed of some different concurrency tools in java
public class LongArrayListUnsafe {
private static final ExecutorService executor
= Executors.newFixedThreadPool(1);
public static void main(String[] args) {
LongArrayList dal1 = new LongArrayList();
int n = 100_000_000;
Timer t = new Timer();
List<Callable<Void>> tasks = new ArrayList<>();
tasks.add(() -> {
for (int i = 0; i <= n; i+=2){
dal1.add(i);
}
return null;
});
tasks.add(() -> {
for (int i = 0; i < n; i++){
dal1.set(i, i + 1);
}
return null;});
tasks.add(() -> {
for (int i = 0; i < n; i++) {
dal1.get(i);
}
return null;});
tasks.add(() -> {
for (int i = n; i < n * 2; i++) {
dal1.add(i + 1);
}
return null;});
try {
executor.invokeAll(tasks);
} catch (InterruptedException exn) {
System.out.println("Interrupted: " + exn);
}
executor.shutdown();
try {
executor.awaitTermination(1000, TimeUnit.MILLISECONDS);
} catch (Exception e){
System.out.println("what?");
}
System.out.println("Using toString(): " + t.check() + " ms");
}
}
class LongArrayList {
// Invariant: 0 <= size <= items.length
private long[] items;
private int size;
public LongArrayList() {
reset();
}
public static LongArrayList withElements(long... initialValues){
LongArrayList list = new LongArrayList();
for (long l : initialValues) list.add( l );
return list;
}
public void reset(){
items = new long[2];
size = 0;
}
// Number of items in the double list
public int size() {
return size;
}
// Return item number i
public long get(int i) {
if (0 <= i && i < size)
return items[i];
else
throw new IndexOutOfBoundsException(String.valueOf(i));
}
// Replace item number i, if any, with x
public long set(int i, long x) {
if (0 <= i && i < size) {
long old = items[i];
items[i] = x;
return old;
} else
throw new IndexOutOfBoundsException(String.valueOf(i));
}
// Add item x to end of list
public LongArrayList add(long x) {
if (size == items.length) {
long[] newItems = new long[items.length * 2];
for (int i=0; i<items.length; i++)
newItems[i] = items[i];
items = newItems;
}
items[size] = x;
size++;
return this;
}
public String toString() {
return Arrays.stream(items, 0,size)
.mapToObj( Long::toString )
.collect(Collectors.joining(", ", "[", "]"));
}
}
public class Timer {
private long start, spent = 0;
public Timer() { play(); }
public double check() { return (System.nanoTime()-start+spent)/1e9; }
public void pause() { spent += System.nanoTime()-start; }
public void play() { start = System.nanoTime(); }
}
The implementation of a LongArrayList class is not so important,it's not threadsafe.
The drivercode with the executorservice performs a bunch of operations on the arraylist, and has 4 different tasks doing it, each 100_000_000 times.
The problem is that when I give the threadpool more threads "Executors.newFixedThreadPool(2);" it only becomes slower.
For example, for one thread, a typical timing is 1.0366974 ms, but if I run it with 3 threads, the time ramps up to 5.7932714 ms.
What is going on? why is more threads so much slower?
EDIT:
To boil the issue down, I made this much simpler drivercode, that has four tasks that simply add elements:
ExecutorService executor
= Executors.newFixedThreadPool(2);
LongArrayList dal1 = new LongArrayList();
int n = 100_000_00;
Timer t = new Timer();
for (int i = 0; i < 4 ; i++){
executor.execute(new Runnable() {
#Override
public void run() {
for (int j = 0; j < n ; j++)
dal1.add(j);
}
});
}
executor.shutdown();
try {
executor.awaitTermination(1000, TimeUnit.MILLISECONDS);
} catch (Exception e){
System.out.println("what?");
}
System.out.println("Using toString(): " + t.check() + " ms");
Here it still does not seem to matter how many threads i allocate, there is no speedup at all, could this simply be because of overhead?
There are some problems with your code that make it hard to reason why with more threads the time increases.
btw
public double check() { return (System.nanoTime()-start+spent)/1e9; }
gives you back seconds not milliseconds, so change this:
System.out.println("Using toString(): " + t.check() + " ms");
to
System.out.println("Using toString(): " + t.check() + "s");
First problem:
LongArrayList dal1 = new LongArrayList();
dal1 is shared among all threads, and those threads are updating that shared variable without any mutual exclusion around it, consequently, leading to race conditions. Moreover, this can also lead to cache invalidation, which can increase your overall execution time.
The other thing is that you may have load balancing problems. You have 4 parallel tasks, but clearly the last one
tasks.add(() -> {
for (int i = n; i < n * 2; i++) {
dal1.add(i + 1);
}
return null;});
is the most computing-intensive task. Even if the 4 tasks run in parallel, without the problems that I have mention (i.e., lack of synchronization around the shared data), the last task will dictate the overall execution time.
Not to mention that parallelism does not come for free, it adds overhead (e.g., scheduling the parallel work and so on), which might be high enough that makes it not worth to parallelize the code in the first place. In your code, there is at least the overhead of waiting for the tasks to be completed, and also the overhead of shutting down the pool of executors.
Another possibility that would also explain why you are not getting ArrayIndexOutOfBoundsException all over the place is that the first 3 tasks are so small that they are being executed by the same thread. This would also again make your overall execution time very dependent on the last task, the on the overhead of executor.shutdown(); and executor.awaitTermination. However, even if that is the case, the order of execution of tasks, and which threads will execute then, is typically non-deterministic, and consequently, is not something that your application should rely upon. Funny enough, when I changed your code to immediately execute the tasks (i.e., executor.execute) I got ArrayIndexOutOfBoundsException all over the place.
I would like to process ranges of integers in rxjava2 and I'm looking for most elegant way to solve it. Brute force solution (processRange function is just a placeholder, in reality it is a lot more complex thing working on from/to variables - you can think about it as a paging thing from database, which needs to be accessed in specific size chunks to be performant):
public static Flowable<Object> processRange(int from, int to) {
return Flowable.just("From:" + from + " to:" + to);
}
public static void main(String[] args) {
int start = 333;
int end = 78777;
int step = 1000;
List<Pair<Integer, Integer>> ranges = Lists.newArrayList();
for (int i = start; i < end; i += step) {
ranges.add(Pair.of(i, Math.min(i + step - 1, end)));
}
Flowable.concat(
ranges.stream()
.map(range -> processRange(range.getLeft(), range.getRight())).collect(Collectors.toList()))
.forEach(System.out::println);
}
I think it is really ugly way of achieving it. I came up with something which reuses a bit of rxjava, but it is still quite verbose and cryptic
int start = 333;
int end = 78777;
int step = 1000;
int count = (end-start-1)/step + 1;
Flowable.concat(
Flowable.range(0,count)
.map(idx -> processRange(start+idx*step,Math.min(start+((idx+1)*step)-1,end))))
.forEach(System.out::println);
Worst case scenario, I will just hide it all inside a method and unit test the hell out of it, but maybe I'm prying an open door here and there is something simpler available?
You could use for example create() operator:
class SO65163142 {
public Flowable<String> processRange(int from, int to) {
return Flowable.just("From:" + from + " to:" + to);
}
public Flowable<String> getRanges(int start, int end, int step) {
return Flowable
.create((FlowableOnSubscribe<Pair<Integer, Integer>>) emitter -> {
int current = start;
while (current + step < end) {
emitter.onNext(new Pair<>(current, current + step - 1));
current += step;
}
emitter.onNext(new Pair<>(current, end));
emitter.onComplete();
}, BackpressureStrategy.BUFFER)
.flatMap(pair -> processRange(pair.first, pair.second));
}
}
Make sure you apply some +/-1 offsets based on your needs. Some verification:
public class SO65163142Test {
#Test
public void getRangesTest() {
SO65163142 tested = new SO65163142();
int start = 333;
int end = 4777;
int step = 1000;
TestSubscriber<String> testSubscriber = tested
.getRanges(start, end, step)
.test();
testSubscriber.assertValues(
"From:333 to:1332",
"From:1333 to:2332",
"From:2333 to:3332",
"From:3333 to:4332",
"From:4333 to:4777"
);
}
}
I am having a bunch of methods that return a CompletableFuture and I would like to chain in a specific way
package com.sandbox;
import java.util.Random;
import java.util.concurrent.CompletableFuture;
import java.util.stream.IntStream;
public class SandboxFutures {
public CompletableFuture<Integer> generateRandom(int min, int max) {
return CompletableFuture.supplyAsync(() -> {
if (min >= max) {
throw new IllegalArgumentException("max must be greater than min");
}
Random r = new Random();
return r.nextInt((max - min) + 1) + min;
});
}
public CompletableFuture<String> printEvenOrOdd(int result) {
return CompletableFuture.supplyAsync(() -> {
if (result % 2 == 0)
return "Even";
else
return "Odd";
});
}
public CompletableFuture<Integer> findFactorial(int evenNumber) {
return CompletableFuture.supplyAsync(() -> {
if (evenNumber <= 0) {
return 0;
}
return IntStream.rangeClosed(2, evenNumber).reduce(1, (x,y) -> x*y);
});
}
public CompletableFuture<Integer> convertToNearestEvenInteger(int oddNumber) {
return CompletableFuture.supplyAsync(() -> {
if (oddNumber <= 0) {
return 2;
}
return oddNumber+1;
});
}
}
I am trying to combine them based on the following rules,
Generate a random number between 1 and 100
If the number is even print Even, if it is odd print Odd
If the number is even call the findFactorial with the random number
If the number is odd find the nearest even via convertToNearestEvenInteger
I am not too clear on how to do the conditional chaining and exception handling. Some examples or code snippets may be helpful.
You can use thenCompose():
CompletableFuture<Integer> n = generateRandom(1, 100)
.thenCompose(i -> printEvenOrOdd(i)
.thenCompose(s -> s.equals("Even")
? findFactorial(i)
: convertToNearestEvenInteger(i)));
System.out.println(n.get());
However, when big even numbers are generated, your factorial method can't store anything bigger than int, so you need to update that.
The way printEvenOrOdd is written makes it more difficult than it needs to be. The problem is that it doesn't print the word "Even" or "Odd", it returns it, which means the original result is lost. The rest of the steps rely on having the actual number. To work around it, you could use call printEvenOrOdd and use .thenApply(__ -> result) to restore the original number afterwards. It would look like this:
System.out.println(
generateRandom(1, 100)
.thenCompose(result ->
printEvenOrOdd(result)
.thenAccept(System.out::println)
.thenApply(__ -> result)
)
.thenCompose(result ->
result % 2 == 0
? findFactorial(result)
: convertToNearestEvenInteger(result)
)
.join()
);
A better solution would be to change the definition of printEvenOrOdd to something like:
public CompletableFuture<Integer> printEvenOrOdd(int result) {
return CompletableFuture.supplyAsync(() -> {
System.out.println(result % 2 == 0 ? "Even" : "Odd");
return result;
});
}
That would make it much easier to chain steps 3 and 4:
System.out.println(
generateRandom(1, 100)
.thenApply(this::printEvenOrOdd)
.thenCompose(result ->
result % 2 == 0
? findFactorial(result)
: convertToNearestEvenInteger(result)
)
.join()
);
I got the code below from a sample code from tutorials point and tweaked it a little bit.
App.java
public static void main(String[] args) throws ParseException {
CommandTest t = new CommandTest();
t.start(args);
}
CommandTest.java
public class CommandTest {
void start(String[] args) throws ParseException {
//***Definition Stage***
// create Options object
Options options = new Options();
// add option "-a"
options.addOption(
Option.builder("a")
.longOpt("add")
.desc("add numbers")
.hasArg(false)
.valueSeparator('=')
.required(false)
.build()
);
// add option "-m"
options.addOption("m", false, "");
options.addOption(
Option.builder("m")
.longOpt("multiply")
.desc("multiply numbers")
.hasArg(false)
.valueSeparator('=')
.required(false)
.build()
);
//***Parsing Stage***
//Create a parser
CommandLineParser parser = new DefaultParser();
//parse the options passed as command line arguments
CommandLine cmd = parser.parse( options, args);
//***Interrogation Stage***
//hasOptions checks if option is present or not
if(cmd.hasOption("a")) {
System.out.println("Sum of the numbers: " + getSum(args));
} else if(cmd.hasOption("m")) {
System.out.println("Multiplication of the numbers: " + getMultiplication(args));
}
}
public static int getSum(String[] args) {
int sum = 0;
for(int i = 1; i < args.length ; i++) {
sum += Integer.parseInt(args[i]);
}
return sum;
}
public static int getMultiplication(String[] args) {
int multiplication = 1;
for(int i = 1; i < args.length ; i++) {
multiplication *= Integer.parseInt(args[i]);
}
return multiplication;
}
}
Now, my question is that, when i try to execute the above code with a parameter of -multi it will still be accepted? I've already set the options to receive only either -m or -multiply. However, it will still accept -multi
I am using commons-cli-1.3.1 (im trying to debug a legacy code by the way)
Note: Above source is a SAMPLE source only, no need to apply actual coding guidelines (good or bad) i just want to know why the behavior happens as it is.
This is the behaviour when a non-matching option gets found (org.apache.commons.cli.Options:233):
public List<String> getMatchingOptions(String opt) {
opt = Util.stripLeadingHyphens(opt);
List<String> matchingOpts = new ArrayList();
if (this.longOpts.keySet().contains(opt)) {
return Collections.singletonList(opt);
} else {
Iterator var3 = this.longOpts.keySet().iterator();
while(var3.hasNext()) {
String longOpt = (String)var3.next();
/******************************************************/
/* longOpt = "multiply" */
/* opt = "multi" */
/******************************************************/
if (longOpt.startsWith(opt)) {
matchingOpts.add(longOpt);
}
/******************************************************/
}
return matchingOpts;
}
}
As you can see in the highlighted block, if a short option isn't matched the library searches for the first long option that partially matches the entered option. It uses startsWith, and since "multiply".startsWith("multi") is true it defaults to option --multiply.
public static void main(String[] args) {
List<String> data = new ArrayList<>();
for (int i = 0; i < 10000000; i++) {
data.add("data" + i);
}
System.out.println("parallel stream start time" + System.currentTimeMillis());
data.parallelStream().forEach(x -> {
System.out.println("data -->" + x);
});
System.out.println("parallel stream end time" + System.currentTimeMillis());
System.out.println("simple stream start time" + System.currentTimeMillis());
data.stream().forEach(x -> {
System.out.println("data -->" + x);
});
System.out.println("simple stream end time" + System.currentTimeMillis());
System.out.println("normal foreach start time" + System.currentTimeMillis());
for (int i = 0; i < data.size(); i++) {
System.out.println("data -->" + data.get(i));
}
System.out.println("normal foreach end time" + System.currentTimeMillis());
}
Output
parallel stream start time 1501944014854
parallel stream end time 1501944014970
simple stream start time 1501944014970
simple stream end time 1501944015036
normal foreach start time 1501944015036
normal foreach end time 1501944015040
Total time taken
Simple stream -> 66
Parellem stream -> 116
simple foreach -> 4
In many blogs written that parallelStream is executing by parallel by internally managed distributed task among thread and collect automatically..
But as per above experiment it is clearly notice that Parallel Stream taking more time then simple stream and normal foreach.
Why it is taking more time if it is executed parallel? Is it good to use in project as this feature is downgrading performance?
Thanks in Advance
Your tests are based on I/O operations (the most expensive operation)
If you want to use parallel streams you have to take the thread creation time overhead into account. So only if your operation benefits from that then use it (that is the case for heavy operations). If not, then just use normal streams or a regular for-loop.
Basic rules for measurement:
Don't use I/O operation.
Repeat the same test more then just once.
So if we have to re-formulate the test scenarios again, then we probably have a test helper class defined as follows:
import java.util.HashMap;
import java.util.Map;
import java.util.UUID;
public class Benchmark {
public static <T> T performTest(Callable<T> callable, int iteration, String name) throws Exception {
Map<String, Iteraion> map = new HashMap<>();
T last = null;
for (int i = 0; i < iteration; i++) {
long s = System.nanoTime();
T temp = callable.call();
long f = System.nanoTime();
map.put(UUID.randomUUID().toString(), new Iteraion(s, f));
if (i == iteration - 1) {
last = temp;
}
}
System.out.print("TEST :\t" + name + "\t\t\t");
System.out.print("ITERATION: " + map.size());
long sum = 0l;
for (String i : map.keySet()) {
sum += (map.get(i).finish - map.get(i).start);
}
long avg = (sum / map.size()) / 1000000;
System.out.println("\t\t\tAVERAGE: " + avg + " ms");
return last;
}
public interface Callable<T> {
T call() throws Exception;
}
static class Iteraion {
Long start;
Long finish;
public Iteraion(Long s, Long f) {
start = s;
finish = f;
}
}
}
Now we can perform the same test more then once using different operation. The following code shows test performed using two different scenarios.
import java.util.ArrayList;
import java.util.List;
import static java.lang.Math.*;
#SuppressWarnings("unused")
public class Test {
public static void main(String[] args) {
try {
final int iteration = 100;
final List<String> data = new ArrayList<>();
for (int i = 0; i < 10000000; i++) {
data.add("data" + i);
}
/**
* Scenario 1
*/
Benchmark.performTest(new Callable<Void>() {
#Override
public Void call() throws Exception {
data.parallelStream().forEach(x -> {
x.trim();
});
return (Void) null;
}
}, iteration, "PARALEL_STREAM_ASSIGN_VAL");
Benchmark.performTest(new Callable<Void>() {
#Override
public Void call() throws Exception {
data.stream().forEach(x -> {
x.trim();
});
return (Void) null;
}
}, iteration, "NORMAL_STREAM_ASSIGN_VAL");
Benchmark.performTest(new Callable<Void>() {
#Override
public Void call() throws Exception {
for (int i = 0; i < data.size(); i++) {
data.get(i).trim();
}
return (Void) null;
}
}, iteration, "NORMAL_FOREACH_ASSIGN_VAL");
/**
* Scenario 2
*/
Benchmark.performTest(new Callable<Void>() {
#Override
public Void call() throws Exception {
data.parallelStream().forEach(x -> {
Integer i = Integer.parseInt(x.substring(4, x.length()));
double d = tan(atan(tan(atan(i))));
});
return (Void) null;
}
}, iteration, "PARALEL_STREAM_COMPUTATION");
Benchmark.performTest(new Callable<Void>() {
#Override
public Void call() throws Exception {
data.stream().forEach(x -> {
Integer i = Integer.parseInt(x.substring(4, x.length()));
double d = tan(atan(tan(atan(i))));
});
return (Void) null;
}
}, iteration, "NORMAL_STREAM_COMPUTATION");
Benchmark.performTest(new Callable<Void>() {
#Override
public Void call() throws Exception {
for (int i = 0; i < data.size(); i++) {
Integer x = Integer.parseInt(data.get(i).substring(4, data.get(i).length()));
double d = tan(atan(tan(atan(x))));
}
return (Void) null;
}
}, iteration, "NORMAL_FOREACH_COMPUTATION");
} catch (Exception e) {
e.printStackTrace();
}
}
}
The first scenario performs the same test using the trim() method 100 times for a list that contains 10_000_000 elements and therefore it uses a parallel stream, then a normal stream and last the old school for loop.
The second scenario performs some relatively heavy operations like tan(atan(tan(atan(i)))) for the same list with the same technique as in the first scenario.
The results are:
// First scenario, average times
Parallel stream: 78 ms
Regular stream: 113 ms
For-loop: 110 ms
// Second scenario, average times
Parallel stream: 1397 ms
Regular stream: 3866 ms
For-loop: 3826 ms
Note that you can debug the above code, then you notice that for parallel streams the program creates three extra threads under name [ForkJoinPool-1], [ForkJoinPool-2] and [ForkJoinPool-3].
Edit:
The sequential streams and the for-loop use the caller's thread.