Develop Reference

.block() within reactive flow (Spring Webflux) for object that holds state

I've created a reactive flow at my controller Endpoint addEntry where one object inside should be created only once per request since it holds a state.
#Override
public Mono<FileResultDto> addEntry(final Flux<byte[]> body,
final String fileId) {
return keyVaultRepository.findByFiletId(fileId)
.switchIfEmpty(Mono.defer(() -> {
final KeyVault keyVault = KeyVault.of(fileId);
return keyVaultRepository.save(keyVault);
}))
.map(keyVault -> Mono
.just(encryption.createEncryption(keyVault.getKey(), ENCRYPT_MODE)) // createEncryption obj. that holds a state
.cache())
.map(encryption -> Flux
.from(body)
.map(bytes -> encryption
.share()
.block()
.update(bytes) // works with the state and changes it per byte[] going through this flux
)
)
.flatMap(flux -> persistenceService.addEntry(flux, fileId));
}
before I asked this question I used
encryption.block() which was failing.
I found this one and updated my code accordingly (added .share()).
The test itself is working. But I am wondering if this is the proper way to go to work with an object that should be created and used only once in the reactive flow, provided by
encryptionService.createEncryption(keyVault.getKey(), ENCRYPT_MODE)
Happy to hear your opinion

Mono.just is only a wrapper around a pre-computed value, so there is no need to cache or share it, because it is only just giving back a cached variable on subscription.
But, in your example, there is something I do not understand.
If we simplify / decompose it, it gives the following:
Mono<KeyVault> vault = keyVaultRepository.findByFiletId(fileId)
.switchIfEmpty(Mono.defer(() -> keyVaultRepository.save(KeyVault.of(fileId));
));
Mono<Mono<Encryption>> fileEncryption = vault
.map(it -> Mono.just(createEncryption(it.getKey)).cache()); // <1>
Mono<Flux<Encryption>> encryptedContent = fileEncryption.map(encryption -> Flux
.from(body)
.map(bytes -> encryption
.share()
.block()
.update(bytes))); // <2>
Mono<FileResultDto> file = encryptedContent.map(flux -> persistenceService.addEntry(flux, fileId));
Why are you trying to wrap your encryption object ? The result is already part of a reactive pipeline. Doing Mono.just() is redundant because you are already in a map operation, and doing cache() over just() is also redundant, because a "Mono.just" is essentially a permanent cache.
What does your "update(bytes)" method do ? Does it mutate the same object every time ? because if it does, you might have a problem here. Reactive streams cannot ensure thread-safety and proper ordering of actions on internal mutated states, that is out of its reach. You might bypass the problem by using scan operator, though.
Without additional details, I would start refactoring the code like this:
Mono<KeyVault> vault = keyVaultRepository.findByFileId(fileId)
.switchIfEmpty(Mono.defer(() -> keyVaultRepository.save(KeyVault.of(fileId));
Mono<Encryption> = vault.map(it -> createEncryption(it.getKey()));
Flux<Encryption> encryptedContent = fileEncryption
.flatMapMany(encryption -> body.scan(encryption, (it, block) -> it.update(block)));
Mono<FileResultDto> result = persistenceService.addEntry(encryptedContent, fileId);

Given an existing Flux (FluxMap), how can I emit message to this Flux?

Our project use a external library. It has a method return FluxMap (since FluxMap is not completely public so just call it Flux):
Flux<MappedType> aFluxMap = Library.createMappingToMappedType();
I have to emit some objects to aFluxMap to get them converted to MappedType (it has private constructor, few setter), then I can:
aFluxMap.doOnNext(converted -> doJob(converted))
I expect that there is a method on Flux/Mono like:
aFluxMap.emit(myObj);
But I could not find any method like that.
I have searched "how to emit to flux dynamically", then there is a solution:
FluxProcessor p = UnicastProcessor.create().serialize();
FluxSink sink = p.sink();
sink.next(mess);
But seem that it emit to newly created flux (p), not my aFluxMap. So I want to ask is there any way to emit message to a existed Flux (or how to connect a FluxSink to a existed Flux, so that whenever FluxSink .next(mess), then the existed Flux get the message). Thank you
Note: please don't pay much attention to the stupidity of the library. We must use it
==========================================
UPDATE:
As #lkatiforis suggestion:
FluxProcessor p = //see above
Flux<MappedType> aFluxMap = Library.createMappingToMappedType();
p.flatMap(raw -> aFluxMap).subscribe();
I got another issue. Library.createMappingToMappedType() return a subscribed Flux with its source is UnicastProcessor (also subscribed).
When I call p.flatMap(raw -> aFluxMap), then internally aFluxMap get subscribed again cause its source also get subscribed again, so that I got an exception telling that "UnicastProcessor can be subscribe once". Any suggestion?

You can create a new stream and then merge the two streams into one by using one of these methods: merge, concat, zip, and their variants.
Here is an example:
Flux<MappedType> yourFlux = //...
Flux<MappedType> aFluxMap = Library.createMappingToMappedType();
Flux.merge(aFluxMap, yourFlux);
The merge operator executes a merging of the MappedType objects from the two provided publisher sequences.

Conversion from Observable to Mono in Kotlin

Trying to insert in couchbase and that has observable return type but want mono,hence did this.
Its compiling but at run time its getting stuck forever at the conversion stage (i.e Mono.from { obs }).
fun saveScopeId(scopeId: ScopeId): Mono<ScopeId> {
val obs = scopeRepository.couchbaseOperations.insert(scopeId)
return Mono.from<ScopeId> { obs }
}

Observable can generate multiple values but if you can guarantee that it will be one item (I assume this is why you want to use Mono here) you can use Mono.fromDirect in this way:
Mono.fromDirect(yourObservable.toFlowable(BackpressureStrategy.BUFFER));
As you can see in example, there is toFlowable method used.
You should see the other backpressure strategies: here

This way we can achieve but not sure about the performance part.
As there Rx -> Rx -> Reactor conversion. Can someone tell me by looking into couchbase SDK 4.x (introduced recently), only if there are some conversion issue.
Mono.fromDirect(RxReactiveStreams.toPublisher(scopeRepository.couchbaseOperations.insert(scope)))

Try this but this thread blocking model.
public Mono<String> GetData(Observable<String> inputData) {
return Mono.fromCallable(() -> inputData.blockingFirst(""));
}

Alternatives to the official (and ugly) Sequential composition

In vertx guides, the sequential composition for chaining async calls is shown below:
FileSystem fs = vertx.fileSystem();
Future<Void> startFuture = Future.future();
Future<Void> fut1 = Future.future();
fs.createFile("/foo", fut1.completer());
fut1.compose(v -> {
// When the file is created (fut1), execute this:
Future<Void> fut2 = Future.future();
fs.writeFile("/foo", Buffer.buffer(), fut2.completer());
return fut2;
}).compose(v -> {
// When the file is written (fut2), execute this:
fs.move("/foo", "/bar", startFuture.completer());
},
// mark startFuture it as failed if any step fails.
startFuture);
Is it just me or is this code really cumbersome and hard to read?
There should be another way without falling into the callback hell.
It's a pity there are so few blog posts on vertx, any ideas are much appreciated.

these days the de facto library for writing asynchronous, non-blocking code on the JVM is RxJava. if you're not familiar i'd say it's highly worth your while to take a look as one of the many benefits is the ability to write "flows" as compositional streams that aren't quite as callback hell-y as the JDK's Futures were.
luckily, Vert.x is integrated with RxJava. as an example, here is your snippet rewritten using RxJava artifacts:
#Override
public void start(Future<Void> startFuture) throws Exception {
final FileSystem fs = vertx.fileSystem();
fs.rxCreateFile("/foo")
.andThen(fs.rxWriteFile("/foo", Buffer.buffer()))
.andThen(fs.rxMove("/foo", "/bar"))
.subscribe(
() -> {
startFuture.complete();
},
error -> {
startFuture.fail(error);
}
);
}
much more concise and readable.
note:
use RxJava 2 as it has superceded RxJava 1
...both versions, however, are supported in Vert.x, with their respective artifacts living in separate namespaces:
io.vertx.rxjava for version RxJava 1 artifacts
io.vertx.reactivex for version RxJava 2 artifacts
hope that helps!

Kafka streams app seperate reads from writes

I am pretty new to Kafka and Kafka Streams so please bear with me. I would like to know if I am on the right track here.
I am writing to a Kafka topic at the moment and try to access the data through a rest service. The raw data kind of needs to be transformed before it will be accessed.
What I have so far is a producer that writes the raw data into a topic.
1.) Now I want streams App (should be a jar running in a container) that just transforms the data in my desired shape. Following the materialized view paradigm here.
Over simplified version of 1.)
KStreamBuilder builder = new KStreamBuilder();
KStream<String, String> source =
builder.stream("my-raw-data-topic");
KafkaStreams streams = new KafkaStreams(builder,props);
KTable<String, Long> t = source.groupByKey().count("My-Table");
streams.start();
2.) And another streams App (should be a jar running in a container) that justs holds the KTable as some sort of Repository which can be accessed via a wrapping rest service.
Here I am kind of stuck with the proper way to work with the api.
What is the bare minimun to access and query a KTable? Do I need to assign the transformation topology to the builder again?
KStreamBuilder builder = new KStreamBuilder();
KTable table = builder.table("My-Table"); //Casting?
KafkaStreams streams = new KafkaStreams(builder, props);
RestService service = new RestService(table);
// Use the Table as Repository which is wrapped by a Rest-Service and gets updated reactivly
Right now this is pseudo code
Am I on the right path here? Does is make sense to separate 1.) and 2.)? Is this the indented way to work with streams to materialize views? For me, it would have the benefit to scale up the writes and the reads via container independently where I see more traffic.
How is the repopulating of the KTable handled on a crash of either 1.) or 2.). Is this done via replication to the streaming api or is this something I would need to address via code. Like resetting the cursor and reply the events?

Couple of comments:
In your code snippet (1) you modify your topology after you handed the builder into the KafkaStreams constructor:
KafkaStreams streams = new KafkaStreams(builder,props);
// don't modify builder anymore!
You should not do this but first specify you topology and afterwards create the KafkaStreams instance.
About splitting you application into two. This can make sense to scale both parts independently. But it's hard to say in general. However, if you do spit both, the first one needs to write the transformed date into an output topic and the second one should read this output topic as a table (builder.table("output-topic-of-transformation") to serve the REST requests.
For accessing the store of the KTable, you need to get a query handle via the provided store name:
ReadOnlyKeyValueStore keyValueStore =
streams.store("My-Table", QueryableStoreTypes.keyValueStore());
See the docs for further details:
http://docs.confluent.io/current/streams/developer-guide.html#interactive-queries