Develop Reference

Kafka Stream exceptions won't trigger commit on the input record

I have a kafka stream that listens on a topic and outputs on another. For ex:
KStream<String, String> messageStream = builder.stream(inputTopic)
KStream<String, String> processedStream = messageStream.process(() -> new CustomProcessor());
processedStream.to(outputTopic)
Inside the process method lets say a NPE occurs. I have implemented an uncaught exception handler that will handle this exception and send the record to dead latter queue. Ofc, this stream will close and after it starts again it will start processing the message again ( because it wasn't commited ). How can i avoid this? How can i commit the record when i receive the exception ?

Everything that happens inside CustomProcessor is under your control. You can wrap your logic into a try/catch and route the results to different streams, e.g. by outputting a "valid" flag and using KStream.split to split the result of your processing into a DLQ and the result.

Adjusting parallism based on number of partitions assigned in Consumer.committablePartitionedSource

I am trying to use Consumer.committablePartitionedSource() and creating stream per partition as shown below
public void setup() {
control = Consumer.committablePartitionedSource(consumerSettings,
Subscriptions.topics("chat").withPartitionAssignmentHandler(new PartitionAssignmentListener()))
.mapAsyncUnordered(Integer.MAX_VALUE, pair -> setupSource(pair, committerSettings))
.toMat(Sink.ignore(), Consumer::createDrainingControl)
.run(Materializer.matFromSystem(actorSystem));
}
private CompletionStage<Done> setupSource(Pair<TopicPartition, Source<ConsumerMessage.CommittableMessage<String, String>, NotUsed>> pair, CommitterSettings committerSettings) {
LOGGER.info("SETTING UP PARTITION-{} SOURCE", pair.first().partition());
return pair.second().mapAsync(16, msg -> CompletableFuture.supplyAsync(() -> consumeMessage(msg), actorSystem.dispatcher())
.thenApply(param -> msg.committableOffset()))
.withAttributes(ActorAttributes.supervisionStrategy(ex -> Supervision.restart()))
.runWith(Committer.sink(committerSettings), Materializer.matFromSystem(actorSystem));
}
While setting up the source per partition I am using parallelism which I want to change based on no of partitions assigned to the node. That I can do that in the first assignment of partitions to the node. But as new nodes join the cluster assigned partitions are revoked and assigned. This time stream not emitting already existing partitions(due to kafka cooperative rebalancing protocol) to reconfigure parallelism.
Here I am sharing the same dispatcher across all sources and if I keep the same parallelism on rebalancing I feel the fair chance to each partition message processing is not possible. Am I correct? Please correct me

If I understand you correctly you want to have a fixed parallelism across dynamically changing number of Sources that come and go as Kafka is rebalancing topic partitions.
Have a look at first example in the Alpakka Kafka documentation here. It can be adjusted to your example like this:
Consumer.DrainingControl<Done> control =
Consumer.committablePartitionedSource(consumerSettings, Subscriptions.topics("chat"))
.wireTap(p -> LOGGER.info("SETTING UP PARTITION-{} SOURCE", p.first().partition()))
.flatMapMerge(Integer.MAX_VALUE, Pair::second)
.mapAsync(
16,
msg -> CompletableFuture
.supplyAsync(() -> consumeMessage(msg),
actorSystem.dispatcher())
.thenApply(param -> msg.committableOffset()))
.withAttributes(
ActorAttributes.supervisionStrategy(
ex -> Supervision.restart()))
.toMat(Committer.sink(committerSettings), Consumer::createDrainingControl)
.run(Materializer.matFromSystem(actorSystem));
So basically the Consumer.committablePartitionedSource() will emit a Source anytime Kafka assigns partition to this consumer and will terminate such Source when previously assigned partition is rebalanced and taken away from this consumer.
The flatMapMerge will take those Sources and merge the messages they output.
All those messages will compete in the mapAsync stage to get processed. The fairness of this competing is really down to the flatMapMerge above that should give equal chance for all the Sources to emit their messages. Regardless of how many Sources are outputing messages, they will all share a fixed parallelism here, which I believe is what you're after.
All those messages eventually get to the Commiter.sink that handles offset committing.

Late outputs missing for Flink's Session Window

In my pipeline's setup I cannot see side outputs for Session Window. I'm using Flink 1.9.1
Version 1.
What I have is this:
messageStream.
.keyBy(tradeKeySelector)
.window(ProcessingTimeSessionWindows.withDynamicGap(new TradeAggregationGapExtractor()))
.sideOutputLateData(lateTradeMessages)
.process(new CumulativeTransactionOperator())
.name("Aggregate Transaction Builder");
lateTradeMessages implementes SessionWindowTimeGapExtractor and returns 5 secodns.
Further I have this:
messageStream.getSideOutput(lateTradeMessages)
.keyBy(tradeKeySelector)
.process(new KeyedProcessFunction<Long, EnrichedMessage, Transaction>() {
#Override
public void processElement(EnrichedMessage value, Context ctx, Collector<Transaction> out) throws Exception {
System.out.println("Process Late messages For Aggregation");
out.collect(new Transaction());
}
})
.name("Process Late messages For Aggregation");
The problem is that I never see "Process Late messages For Aggregation" when I'm sending messages with same key that should miss window time.
When Session Window passes and I "immediately" sent a new message for the same key it triggers new Session Window without going into Late SideOutput.
Not sure What I'm doing wrong here.
What I would like to achieve here, is to catch "late events" and try to
reprocess them.
I will appreciate any help.
Version 2, after #Dominik Wosiński comment:
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setRestartStrategy(RestartStrategies.fixedDelayRestart(1000, 1000));
env.setParallelism(1);
env.disableOperatorChaining();
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
env.getConfig().setAutoWatermarkInterval(1000);
DataStream<RawMessage> rawBusinessTransaction = env
.addSource(new FlinkKafkaConsumer<>("business",
new JSONKeyValueDeserializationSchema(false), properties))
.map(new KafkaTransactionObjectMapOperator())
.assignTimestampsAndWatermarks(new AssignerWithPeriodicWatermarks<RawMessage>() {
#Nullable
#Override
public Watermark getCurrentWatermark() {
return new Watermark(System.currentTimeMillis());
}
#Override
public long extractTimestamp(RawMessage element, long previousElementTimestamp) {
return element.messageCreationTime;
}
})
.name("Kafka Transaction Raw Data Source.");
messageStream
.keyBy(tradeKeySelector)
.window(EventTimeSessionWindows.withDynamicGap(new TradeAggregationGapExtractor()))
.sideOutputLateData(lateTradeMessages)
.process(new CumulativeTransactionOperator())
.name("Aggregate Transaction Builder");
Watermarks are progressing, I've checked in Flink's Metrics. The Window operator is execution, but still there are no Late Outputs.
BTW, Kafka topic can be idle, so I have to emit new WaterMarks periodically.

The watermark approach looks very suspicious to me. Usually, you would output the latest event timestamp at this point.
Just some background information, so that it's easier to understand.
Late events refer to events that come after the watermark processed to a time after the event. Consider the following example:
event1 #time 1
event2 #time 2
watermark1 #time 3
event3 #time 1 <-- late event
event4 #time 4
Your watermark approach would pretty much render all past events as late events (a bit of tolerance because of the 1s watermark interval). This would also make reprocessing and catchups impossible.
However, you are actually not seeing any late events which is even more surprising to me. Can you double-check your watermark approach, describe your use case, and provide example data? Often times, the implementation is not ideal for the actual use case and it should be solved in a different way.

You are using ProcessingTime in Your case, this means that the system time is used to measure the flow of the time in the DataStream.
For each event, the timestamp assigned to this event is the moment that You receive the data in Your Flink Pipeline. This means that there is no way to have events out-of-order for Flink processing time. Because of that, You will never have late elements for Your windows.
If You switch to EventTime, then for proper input data You should be able to see the late elements being passed to side output.
You probably should take look at the documentation, where there are various concepts of time in Flink explained.

KStream-KTable Inner Join Lost Messages with Exactly Once Config

When I do not set processing.gurantee which means stream will be started with its default value (at_least_once), this code can log successfully and send joined messages to relevant topic.
When the exactly_once config is enabled on this same stream application, some of the data are not able to pass through join successfully. Even there are logs for the first peek block, I can not see some of the second peek logs and some of the messages that I need to have.
I'm sure that both kstream and ktable have required values to be which are not null. And both side gets messages regularly.
Stream Configs:
processing.guarantee=exactly_once
replication.factor=3 (this increases replication factor for internal topics)
Kafka (with 3 broker) Details:
version=2.2.0
log.roll.ms=3600000
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3
message.max.bytes=2000024
Question is, How exactly_once processing guarantee setting can cause this kind of situation?
final KStream<String, UserProfile> userProfileStream = builder.stream(TOPIC_USER_PROFILE);
final KTable<String, Device> deviceKTable = builder.table(TOPIC_DEVICE);
userProfileStream
.peek((genericId, userProfile) ->
log.debug("[{}] Processing user profile: {}", openUserId, userProfile)
)
.join(
deviceKTable,
(userProfile, device) -> {
userProfile.setDevice(device);
return userProfile;
},
Joined.with(Serdes.String(), userProfileSerde, deviceSerde)
)
.peek((genericId, userProfile) ->
log.debug("[{}] Updated user profile: {}", genericId, userProfile)
)
.to(TOPIC_UPDATED_USER_PROFILE, Produced.with(Serdes.String(), userProfileSerde));

How not to depend on subscription time in Reactor

I've been reading throughout the Reactor documentation, but I was not being able to find proper pattern for the following problem.
I have a method that is supposed to do something asynchronously. I returns the result responses in form of a Flux and the consumer could subscribe to it.
The method has following definition:
Flux<ResultMessage> sendRequest(RequestMessage message);
The returning flux is a hot flux, results can come at any given time asynchronously.
The potential consumer could use it in following manner:
sendRequest(message).subscribe(response->doSomethinWithResponse(response);
An implementation can be like this:
Flux<ResultMessage> sendRequest(RequestMessage message) {
Flux<ResultMessage> result = incomingMessageStream
.filter( resultMessage -> Objects.equals( resultMessage.getId(), message.getId() ) )
.take( 2 );
// The message sending is done here...
return result;
}
Where the incomingMessageStream is a Flux of all messages going through this channel.
Problem with this implementation is that consumer is subscribed after the result messages are coming, and it can miss some of them.
So, what I am looking for is a solution that will allow consumer not to depend on time of subscription. A potential consumer may not be required to subscribe to resulting Flux at all. I am looking for a general solution, but if it is not possible you can assume that number of resulting messages is not greater than 2.

After some time I created a solution that seems to work:
Flux<ResultMessage> sendRequest(RequestMessage message) {
final int maxResponsesCount = 2;
final Duration responseTimeout = Duration.ofSeconds( 10 );
final Duration subscriptionTimeout = Duration.ofSeconds( 5 );
// (1)
ConnectableFlux<ResultMessage> result = incomingMessageStream
.ofType( ResultMessage.class )
.filter( resultMessage ->Objects.equals(resultMessage.getId(), message.getId() ) )
.take( maxResponsesCount )
.timeout( responseTimeout )
.replay( maxResponsesCount );
Disposable connectionDisposable = result.connect();
// (2)
AtomicReference<Subscription> subscriptionForCancelSubscription = new AtomicReference<>();
Mono.delay( subscriptionTimeout )
.doOnSubscribe( subscriptionForCancelSubscription::set )
.subscribe( x -> connectionDisposable.dispose() );
// The message sending is done here...
// (3)
return result
.doOnSubscribe(s ->subscriptionForCancelSubscription.get().cancel())
.doFinally( signalType -> connectionDisposable.dispose() );
}
I am using a ConnectableFlux that connects to the stream immediately, without subscribing, which is set to use reply() method to store all messages, so any subscriber at the later point would not miss response messages (1).
There are few paths this can be executed:
Method is called and no subscription has performed on the flux
Solution - there is a timer that removes the connected flux resource after 5 seconds if no subscription is done. (2)
Method is called and subscribed to the flux
2.1. No message has been returned
Solution - there is a timeout set for getting responses (.timeout( responseTimeout )). After that .doFinally(..) cleans the resources (1)(3).
2.2. Some of response messages have been returned
Solution - same as 2.1.
2.3. All response messages have been returned
Solution - The doFinally() is executed due to max number of elements reached ( .take( maxResponsesCount ) ) (1)(3)
I am yet to perform some serious testing on this, if something goes wrong, I'll add the correction to this answer.