Develop Reference

how can I run AKKA StreamRefs sequentially?

I'm using AKKA streamRefs (version 2.5.32), with java API.
I've tried to build a graph (using graphDSL or the PartitionHub classes) in which I can stream some data in a Round-Robin manner and upon its completion broadcasting all the servers a 'FINISH' message,
identifing the stream completion using monitorMat and watchTermination didn't work for me.
I used the AKKA documentation to build a graph in which I stream to multiple sinks some data as follows:
(using watchTermination)
RunnableGraph<Source<EventAccessCollection, NotUsed>> runnableGraph =
source.toMat(PartitionHub.ofStateful(EventAccessCollection.class, () -> new RoundRobin<EventAccessCollection>(),
workersSinks.size()), Keep.right());
Source<EventAccessCollection, NotUsed> fromProducer = runnableGraph.run(mat);
for (Sink<EventAccessCollection, NotUsed> w:workersSinks) {
fromProducer.runWith(w,mat);
}
fromProducer.watchTermination((
(mat,completionStage)->
completionStage.toCompletableFuture().thenRun(() -> {
broadcastMessage(FINISH);
})
));
The documentation can be found here:
https://doc.akka.io/docs/akka/2.5.32/stream/stream-dynamic.html
It seems that since SinkRef must be of the type <T,NotUsed> I can't use it to get feedback about the stream (as opposed to CompletionStage<Done>)
Another thought I had in mind was that maybe streaming/sending messages from different graphs can be defined sequentially so they will arrive to the recipents in the order in which they were sent.
How can I force order on the streams/messages which I want to send to the recipients?

Wait for reactive change stream subscription to be active with Spring Data MongoDB?

When subscribing to change streams using the blocking Spring Data Mongo implementation one can call await to wait for a subscription to become active:
Subscription subscription = startBlockingMongoChangeStream();
subscription.await(Duration.of(2, SECONDS));
Document someDocument = ..
writeDocumentToMongoDb(someDocument);
The startBlockingMongoChangeStream is implemented along these lines:
public Subscription startBlockingMongoChangeStream() {
MessageListenerContainer container = new DefaultMessageListenerContainer(template);
container.start();
MessageListener<ChangeStreamDocument<Document>, Document> listener = System.out::println;
ChangeStreamRequestOptions options = new ChangeStreamRequestOptions("user", ChangeStreamOptions.empty());
return container.register(new ChangeStreamRequest<>(listener, options), Document.class);
}
If await is not used in the example above there's a chance (virtually 100% chance if the JVM is hot) that someDocument is written before the subscription is active and thus the someDocument is missed. So adding await mitigates this issue.
I'm looking for a way to achieve the same thing when using the reactive implementation. The code now looks something like this:
Disposable disposable = startReactiveMongoChangeStream().subscribe(); // (1)
Document someDocument = ..
writeDocumentToMongoDb(someDocument).subscribe(); // (2)
The problem here is, again, that someDocument is written before the subscription returned by startReactiveMongoChangeStream has started and thus the document is missed.
Also note that this is a somewhat contrived example since in my actually application writeDocumentToMongoDb (2) is not aware of the startReactiveMongoChangeStream subscription (1) so I cannot simply flatMap (1) and call (2). The startReactiveMongoChangeStream method is implemented along these lines:
public Flux<ChangeStreamEvent<String>> startReactiveMongoChangeStream() {
return reactiveTemplate.changeStream(String.class)
.watchCollection("user")
.listen();
}
How can I "simulate" the await functionality available in the blocking implementation in the reactive implementation?

TL;DR
There are no means for synchronization in the reactive API
Explanation
First, let's look at both implementations to understand why this is.
The blocking implementation uses MongoDB's cursor API to obtain a cursor. Obtaining a cursor includes a conversation with the server. After MessageListenerContainer has obtained the cursors, it switches the subscription task to active which means that you have awaited the stage where the first cursor was fetched.
The reactive implementation operates on a ChangeStreamPublisher. From the reactive streams protocol, one can get notified when an element is emitted, when the stream completes or fails. There's no notification available when the server-side activity starts or completes. Therefore, you cannot wait until the reactive API receives the first cursor. Since cursors may be empty, the first cursor might not emit any value at all.
I think the MongoDB driver could provide a callback-style API to get notified that the stream is active. That's however something to report in the MongoDB issue tracker.

Java/Quarkus Kafka Streams Reading/Writing to Same Topic based on a condition

Hello I have this issue that I'm trying to solve. Basically I have a Kafka Streams topology that will read JSON messages from a Kafka topic and that message gets deserialized into a POJO. Then ideally it will read check that message for a certain boolean flag. If that flag is true it will do some transformation and then write it back to the topic. However if the flag is false, I'm trying to have it not write anything but I'm not sure how I can go about it. With the MP Reactive Messaging I can just use an RxJava 2 Flowable Stream and return something like Flowable.empty() but I can't use that method here it seems.
JsonbSerde<FinancialMessage> financialMessageSerde = new JsonbSerde<>(FinancialMessage.class);
StreamsBuilder builder = new StreamsBuilder();
builder.stream(
TOPIC_NAME,
Consumed.with(Serdes.Integer(), financialMessageSerde)
)
.mapValues (
message -> checkCondition(message)
)
.to (
TOPIC_NAME,
Produced.with(Serdes.Integer(), financialMessageSerde)
);
The below is the function call logic.
public FinancialMessage checkCondition(FinancialMessage rawMessage) {
FinancialMessage receivedMessage = rawMessage;
if (receivedMessage.compliance_services) {
receivedMessage.compliance_services = false;
return receivedMessage;
}
else return null;
}
If the boolean is false it just returns a JSON body with "null".
I've tried changing the return type of the checkCondition function wrapped like
public Flowable<FinancialMessage> checkCondition (FinancialMessage rawMessage)
And then having the return from the if be like Flowable.just(receivedMessage) or Flowable.empty() but I can't seem to serialize the Flowable object. This might be a silly question but is there a better way to go about this?

Note that Kafka messages are immutable and not deleted after read, and if you read/write from the same topic with a single application, a message would be processed infinitely often (or to be more precise different copies of it) if you don't have a condition to "break" the cycle.
Also, if for example 5 services read from the same topic, all 5 services get a copy of every event. And if one service write back, all other 4 services and the writing service itself will read the message again. Thus, you get quite some data amplification.
If you have different services to react on the original input message consecutively, you could have one topic between each pair of consecutive services to really build a pipeline though.
Last, you say if the boolean flag is true you want to transform the message and emit (I assume for the next service to consumer). And for false you want to do nothing. I a further assume that for a message only a single flag will be true and a successful transformation also switches the flag (to enable processing by the next service). For this case, it's best if you can ensure that each original input message has the same initial boolean flag set to build your pipeline. Thus, only the corresponding service will read messages with its boolean flag set (you don't even need to check the boolean flag as your upstream write ensures that it's set; you could only have a sanity check).
If you don't know which boolean flag is set initially and all services read from the same input topic, just filtering out the message is correct. If all services read all messages, 4 services will filter the message while one service will process it and emit a new message with a different flag. For this architecture, a single topic might work: if a message is processed by all services and all boolean flags are false (after all services processed the message), and you write it back to the input topic, all services would drop the last copy correctly. However, using a single topic implies a lot of redundant reading/writing.
Maybe the best architecture is, to have your original input topic, and one additional input topic for each service. You also use an additional "dispatcher" service that read from the original input topics, and branches() the KStream into the service input topics according to the boolean flag. This way, each service will read only messages with the right flag set to true. Furthermore, each service will write to the input topic of the other services also using branch() after the message transformation to write it to the input topic of the correct next service. Last, you would want an output topic that each service can write into after a message is fully processed.

Conditional logic on a Reactor Flux

I am a Reactor newbie. I am trying to develop the following application logic:
Read messages from a Kafka topic source.
Transform the massages.
Write a subset of the transformed messages to a new Kafka topic target.
Explicitly acknowledge the reading operation for all the messages originally read from topic source.
The only solution I found is to rewrite the above business logic as it follows.
Read messages from a Kafka topic source.
Transform the massages.
Immediately acknowledge the message not be written to topic target.
Filter all the above messages.
Write the rest of the transformed messages to the new Kafka topic target.
Explicitly acknowledge the reading operation for these messages
The code implementing the second logic is the following:
receiver.receive()
.flatMap(this::processMessage)
.map(this::acknowledgeMessagesNotToWriteInKafka)
.filter(this::isMessageToWriteInKafka)
.as(this::sendToKafka)
.doOnNext(r -> r.correlationMetadata().acknowledge());
Clearly, receiver type is KafkaReceiver, and method sendToKafka uses a KafkaSender. One of the things I don't like is that I am using a map to acknowledge some messages.
Is there any better solution to implement the original logic?

This is not exactly your four business logic steps, but I think it's a little bit closer to what you want.
You could acknowledge the "discarded" messages that won't be written in .doOnDiscard after .filter...
receiver.receive()
.flatMap(this::processMessage)
.filter(this::isMessageToWriteInKafka)
.doOnDiscard(ReceiverRecord.class, record -> record.receiverOffset().acknowledge())
.as(this::sendToKafka)
.doOnNext(r -> r.correlationMetadata().acknowledge());
Note: you'll need to use the proper object type that was discarded. I don't know what type of object the Publisher returned from processMessage emits, but I assume you can get the ReceiverRecord or ReceiverOffset from it in order to acknowledge it.
Alternatively, you could combine filter/doOnDiscard into a single .handle operator...
receiver.receive()
.flatMap(this::processMessage)
.handle((m, sink) -> {
if (isMessageToWriteInKafka(m)) {
sink.next(m);
} else {
m.getReceiverRecord().getReceiverOffset().acknowledge();
}
})
.as(this::sendToKafka)
.doOnNext(r -> r.correlationMetadata().acknowledge());

How to skip certain handlers and directly go to specific handler in netty

Let's say I have these handler flow in netty pipeline:
UpHandler1 -> UpHandler2 -> UpHandler3 -> ... -> DownHandler1 -> DownHandler2 -> DownHandler3
Based on certain condition (i.e. already found response to request without doing further processing), is there anyway, in my UpHandler2, I can straight go to DownHandler2 (so skip certain upstream as well as downstream handlers in between)? Is this recommended?

You can use UpHandler2's ChannelHandlerContext to retrieve the ChannelPipeline. From here you can retrieve the channel handler context of any channel handler using one of the context(...) methods. Then sendDownstream for Netty 3, or write for Netty 4, will forward to the next downstream handler after the handler to which the context responds. In effect I think you'll need to get the ChannelHandlerContext for DownHandler1 and use that to write your message.
Alternatively you can build the netty pipeline such that DownHandler2 is the next down stream handler from UpHandler2. If I've understood your pipeline correctly then something like
pipeline.addLast("down3", downhandler3);
pipeline.addLast('up1", uphandler1);
pipeline.addLast("down2", downhandler2);
pipeline.addLast("up2", uphandler2);
pipeline.addLast("down1", downhandler1);
pipeline.addLast("up3", uphandler3);
might work. However this could be quite brittle and also depends on whether your processing logic allows it.