I'm using AKKA streamRefs (version 2.5.32), with java API.
I've tried to build a graph (using graphDSL or the PartitionHub classes) in which I can stream some data in a Round-Robin manner and upon its completion broadcasting all the servers a 'FINISH' message,
identifing the stream completion using monitorMat and watchTermination didn't work for me.
I used the AKKA documentation to build a graph in which I stream to multiple sinks some data as follows:
(using watchTermination)
RunnableGraph<Source<EventAccessCollection, NotUsed>> runnableGraph =
source.toMat(PartitionHub.ofStateful(EventAccessCollection.class, () -> new RoundRobin<EventAccessCollection>(),
workersSinks.size()), Keep.right());
Source<EventAccessCollection, NotUsed> fromProducer = runnableGraph.run(mat);
for (Sink<EventAccessCollection, NotUsed> w:workersSinks) {
fromProducer.runWith(w,mat);
}
fromProducer.watchTermination((
(mat,completionStage)->
completionStage.toCompletableFuture().thenRun(() -> {
broadcastMessage(FINISH);
})
));
The documentation can be found here:
https://doc.akka.io/docs/akka/2.5.32/stream/stream-dynamic.html
It seems that since SinkRef must be of the type <T,NotUsed> I can't use it to get feedback about the stream (as opposed to CompletionStage<Done>)
Another thought I had in mind was that maybe streaming/sending messages from different graphs can be defined sequentially so they will arrive to the recipents in the order in which they were sent.
How can I force order on the streams/messages which I want to send to the recipients?
Related
From the examples I have seen the below code snippet and it works fine. But the problem is that : I don't always have requirements of processing the input-stream and produce it to a sink.
What if I have an application where based on some events I have to only publish to a kafka topic so that down-stream applications can make certain decisions. That means, I don't really have an input-stream but I just know when something happens in my application, I need to publish a message to a particular topic of kafka. That is, I only need a sink.
I was going through examples but didn't find anything matching to my requirements. Is there a way to only configure a KafkaSink that exposes a method() to be called for publishing messages to a topic.
Many thanks in advance!!
String inputTopic = "flink_input";
String outputTopic = "flink_output";
String consumerGroup = "baeldung";
String address = "localhost:9092";
StreamExecutionEnvironment environment = StreamExecutionEnvironment
.getExecutionEnvironment();
FlinkKafkaConsumer011<String> flinkKafkaConsumer = createStringConsumerForTopic(
inputTopic, address, consumerGroup);
DataStream<String> stringInputStream = environment
.addSource(flinkKafkaConsumer);
FlinkKafkaProducer011<String> flinkKafkaProducer = createStringProducer(
outputTopic, address);
stringInputStream
.map(new WordsCapitalizer())
.addSink(flinkKafkaProducer);
You must have a source. You might want to implement a custom source, or you could use something like a NumberSequenceSource followed by an operator like a process function that emits whatever you know you want to write to the sink, followed by the sink.
That process function could, for example, transform the incoming events into whatever you want to write to Kafka, or it could ignore its inputs and use a timer to generate the events to be sent to Kafka.
Or you might find that async i/o is a better building block than a process function, depending on your requirements.
Just getting my head around Mutiny API (and java stream api)...
I have the following code that reads messages off an AWS SQS queue, ref: quarkus sqs guide
Uni<List<Quark>> result =Uni.createFrom()
.completionStage(sqs.receiveMessage(m -> m.maxNumberOfMessages(10).queueUrl(queueUrl)))
.onItem().transform(ReceiveMessageResponse::messages)
.onItem().transform(m -> m.stream().map(Message::body).map(this::toQuark).collect(Collectors.toList()));
Next I want to send each element in the list to a method handleMessage(Quark quark). How do I do this in a "mutiny way!". Do i need to transform again or should not collect..or ?
At the moment, you get a Uni<List<Quark>>. The Mutiny way would be to transform this into a Multi and process each item:
Multi<Quark> multi = result.onItem().transformToMulti(list -> Multi.createFrom().items(list));
A Multi is a stream. Each item will be a Quark. Then, you just need to do the following:
multi.onItem().invoke(q -> handleMessage(q))
I used invoke because I don't know what handleMessage is doing. If it's processing the Quark and returning something, use transform. If it does not return anything, use invoke.
BTW, do not forget to subscribe to the returned Multi.
Update!
I have fixed minor bugs in the sample code after solving some of the problems that were irrelevant to the main question which is still about non-blocking streaming between services.
Background info:
I'm porting a Spring WebFlux service under Quarkus. The service runs long searches on multiple huge data sets and returns the partial results in a Flux (text/event-stream) as they become available.
Problem:
Right now I'm trying to use Mutiny Multi with Vert.x under Quarkus but cannot figure out how the consumer service could receive this stream without blocking.
In all examples the consumer is either a JS front end page or the producer's content type is application/json that seems to bluck until the Multi completes before sending it over in one JSON object (which makes no sense in my application).
Questions:
How to receive text/event-stream with the Mutiny-flavoured Vert.x WebClient?
If the problem would be that the WebClient is not able to receive continuous steams: What is the standard way to stream data between two Quarkus services?
Here is a simplified example
Test entity
public class SearchResult implements Serializable {
private String content;
public SearchResult(String content) {
this.content = content;
}
//.. toString, getters and setters
}
Producer 1. simple infinite stream -> hangs
#GET
#Path("/search")
#Produces(MediaType.SERVER_SENT_EVENTS)
#SseElementType(MediaType.APPLICATION_JSON)
public Multi<SearchResult> getResults() {
return Multi.createFrom().ticks().every(Duration.ofSeconds(2) .onItem().transform(n -> new SearchResult(n.toString()));
}
Producer 2. with Vertx Paths infinite stream -> hangs
#Route(path = "/routed", methods = HttpMethod.GET)
public Multi<SearchResult> getSrStreamRouted(RoutingContext context) {
log.info("routed run");
return ReactiveRoutes.asEventStream(Multi.createFrom().ticks().every(Duration.ofSeconds(2))
.onItem().transform(n -> new SearchResult(n.toString()));
}
Producer 3. simple finite stream -> blocks until completion
#GET
#Path("/search")
#Produces(MediaType.SERVER_SENT_EVENTS)
#SseElementType(MediaType.APPLICATION_JSON)
public Multi<SearchResult> getResults() {
return Multi.createFrom().ticks().every(Duration.ofSeconds(2))
.transform().byTakingFirstItems(5)
.onItem().transform(n -> new SearchResult(n.toString()));
}
Consumer:
Tried multiple different solutions on both producer and consumer sides, but in every case the the stream blocks until it is complete or hangs indefinitely without transferring data for infinite streams. I get the same results with httpie. Here is the latest iteration:
WebClientOptions webClientOptions = new WebClientOptions().setDefaultHost("localhost").setDefaultPort(8182);
WebClient client = WebClient.create(vertx, webClientOptions);
client.get("/string")
.send()
.onFailure().invoke(resp -> log.error("error: " + resp))
.onItem().invoke(resp -> log.info("result: " + resp.statusCode()))
.toMulti()
.subscribe().with(r -> log.info(String.format("Subscribe: code:%d body:%s",r.statusCode(), r.bodyAsString())));
The Vert.x Web Client does not work with SSE (Without configuration).
From https://vertx.io/docs/vertx-web-client/java/:
Responses are fully buffered, use BodyCodec.pipe to pipe the response to a write stream
It waits until the response completes. You can either use the raw Vert.x HTTP Client or use the pipe codec. Examples are given on https://vertx.io/docs/vertx-web-client/java/#_decoding_responses.
Alternatively, you can use an SSE client such as in:
https://github.com/quarkusio/quarkus-quickstarts/blob/master/kafka-quickstart/src/test/java/org/acme/kafka/PriceResourceTest.java#L27-L34
I am a Reactor newbie. I am trying to develop the following application logic:
Read messages from a Kafka topic source.
Transform the massages.
Write a subset of the transformed messages to a new Kafka topic target.
Explicitly acknowledge the reading operation for all the messages originally read from topic source.
The only solution I found is to rewrite the above business logic as it follows.
Read messages from a Kafka topic source.
Transform the massages.
Immediately acknowledge the message not be written to topic target.
Filter all the above messages.
Write the rest of the transformed messages to the new Kafka topic target.
Explicitly acknowledge the reading operation for these messages
The code implementing the second logic is the following:
receiver.receive()
.flatMap(this::processMessage)
.map(this::acknowledgeMessagesNotToWriteInKafka)
.filter(this::isMessageToWriteInKafka)
.as(this::sendToKafka)
.doOnNext(r -> r.correlationMetadata().acknowledge());
Clearly, receiver type is KafkaReceiver, and method sendToKafka uses a KafkaSender. One of the things I don't like is that I am using a map to acknowledge some messages.
Is there any better solution to implement the original logic?
This is not exactly your four business logic steps, but I think it's a little bit closer to what you want.
You could acknowledge the "discarded" messages that won't be written in .doOnDiscard after .filter...
receiver.receive()
.flatMap(this::processMessage)
.filter(this::isMessageToWriteInKafka)
.doOnDiscard(ReceiverRecord.class, record -> record.receiverOffset().acknowledge())
.as(this::sendToKafka)
.doOnNext(r -> r.correlationMetadata().acknowledge());
Note: you'll need to use the proper object type that was discarded. I don't know what type of object the Publisher returned from processMessage emits, but I assume you can get the ReceiverRecord or ReceiverOffset from it in order to acknowledge it.
Alternatively, you could combine filter/doOnDiscard into a single .handle operator...
receiver.receive()
.flatMap(this::processMessage)
.handle((m, sink) -> {
if (isMessageToWriteInKafka(m)) {
sink.next(m);
} else {
m.getReceiverRecord().getReceiverOffset().acknowledge();
}
})
.as(this::sendToKafka)
.doOnNext(r -> r.correlationMetadata().acknowledge());
I'm currently working with Akka Streams (in Java) for a personal project and I'm having a hard time understanding how to send element to a Source.
The idea is to use a WebSocket to push content into the user's web browser. I've managed to use Akka Streams to create a request-response system, following the Akka HTTP documentation, but this is not what I want to do.
Looking into the Akka Streams documentation, I saw that there is Source.queue and Source.actorRef. But I don't understand how to put an element into the Source. Source.queue and Source.actorRef return a Source, which doesn't have the method offer (for Source.queue) or tell (for Source.actorRef).
My question is: how do I get the ActorRef for the Source created by Source.actorRef or the SourceQueueWithComplete for a Source created with Source.queue, to be able to send elements to my Source?
I searched the various Akka documentation but found no method to do that. And the majority of the code I found on the Internet is written in Scala, which doesn't seem to have the same problem.
The actor and queue from Source.actorRef and Source.queue, respectively, are the materialized values of those sources, meaning that they can be obtained only if the stream is running. For example:
final ActorRef actor =
Source.actorRef(Integer.MAX_VALUE, OverflowStrategy.fail())
.to(Sink.foreach(m -> System.out.println(m)))
.run(materializer);
actor.tell("do something", ActorRef.noSender());
It's no different in Scala:
implicit val materializer = ActorMaterializer()
val actor =
Source.actorRef(Int.MaxValue, OverflowStrategy.fail)
.to(Sink.foreach(println))
.run()
actor ! "do something"