I'm using a spring boot app with kafka integration and I want to implement an endpoint to stop and start kafka from publishing messages.
The message are triggered in a async way by another endpoints.
The beans KafkaTemplate<String, String> or ProducerFactory<String, String> producerFactory() does not have any stop and pause actions.
My goal is to be able to simulate connection failures and make sure those message are stored in a fallback mechanisms that I have in place.
Any ideas?
The KafkaTemplate doesn't have those callbacks because it is a passive component which can do the stuff only if we call it.
For the simulation of the connection failure, I would suggest you to implement a custom ProducerFactory producing a KafkaProducer with mocked or overridden Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback);. There, in that ProducerFactory you can implement appropriate lifecylce callbacks and react to the state in the mentioned send() implementation.
The org.apache.kafka.clients.producer.MockProducer may have something for your to reuse or borrow. For example see its close() or fenceProducer().
Related
I am working on a microservices project. In this project, there is a Microservice A is doing a process in various steps. At the completion of each step, Microservice sends a message into a kafka topic. Then another Microservice B consumes the message from the kafka topic and sends an email notifying the successful completion of the step. I need Exactly once semantics for this. I am using KafkaTemplate.send in Microservice A and #KafkaListener to read the message in Microservice B. My question is whether KafkaTemplate producer and #KafkaListener consumer are idempotent and if not, how can I make them idempotent.
Regards,
I am creating autowiring the KafkaTemplate using the following code:-
#Autowired
public EventProducer(NewTopic topic, KafkaTemplate<String, Event> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
Exactly once semantics in Kafka apply to consume->process->produce operations within the same application - even then, only the entire cpp is "exactly once"; the consume->process part is at least once; consumption is always at least once (or at most once), including in your scenario (for B).
I work with apache-kafka and web flux (spring boot) and I want to know if there is a method to auto create a KafkaListener for each topic I add in application.yml(or properties)
This is not what consumer is for. The Kafka topic is a stream of data constantly changing . What is business purpose of that http request? Maybe you want to stream such a topic request to the Flux? Then consider to use Spring Integration dynamic flows and its toReactivePublisher() feature:
https://docs.spring.io/spring-integration/docs/current/reference/html/dsl.html#java-dsl-runtime-flows
https://docs.spring.io/spring-integration/docs/current/reference/html/reactive-streams.html#java-dsl
This sample shows something about Kafka and dynamic flows: https://github.com/spring-projects/spring-integration-samples/tree/main/dsl/kafka-dsl.
Also this one demonstrates some “to WebFlux” technique : https://github.com/artembilan/sandbox/tree/master/amqp-to-webflux.
Or you can look into Reactor Kafka: https://projectreactor.io/docs/kafka/release/reference/.
We have a requirement where we are consuming messages from one topic then there is some enrichment happening and then we are publishing the message to another topic. below are the events
Consumer - Consume the message
Enrichment - Enriched the consumed message
Producer - Published Enriched message to other topic
I am using Spring cloud kafka binder and things are working fine. suddenly we observed that producer is sending duplicate message to the topic and then we made Producer is idempotent. We have autocommitOffSet to false for better control. Below is what we are doing in the method
#StreamListener("INPUT")
#SendTo("OUTPUT")
public void consumer(Message message){
String inputMessage = message.getPayload.toString();
String enrichMessage = // Enrichment on inputMessage
return enrichMessage;
}
We observed if ack.acknowledge() failed due to some issue, Message still sent to the outbound channel. How can we handle entire consumer/producer as part of one transaction so that if acknowledge fail message will not sent to the topic.
I have set below transaction properties as well
spring.cloud.stream.kafka.binder.transaction.transactionIdPrefix=TX-
spring.cloud.stream.kafka.binder.transaction.producer.configuration.ack=all
spring.cloud.stream.kafka.binder.transaction.producer.configuration.retries=1
spring.cloud.stream.kafka.bindings.input.consumer.autoCommitOffset=true
spring.cloud.stream.kafka.bindings.input.consumer.enableDlq=true
spring.cloud.stream.kafka.bindings.input.consumer.dlqName=error.topic
spring.cloud.stream.kafka.bindings.input.consumer.autoCommitOnError=true
If there is any example available that would be really helpful.
Cheers
You need to make the binder transactional. See the documentation
https://docs.spring.io/spring-cloud-stream-binder-kafka/docs/3.1.4/reference/html/spring-cloud-stream-binder-kafka.html#_kafka_binder_properties
spring.cloud.stream.kafka.binder.transaction.transactionIdPrefix
Enables transactions in the binder. See transaction.id in the Kafka documentation and Transactions in the spring-kafka documentation. When transactions are enabled, individual producer properties are ignored and all producers use the spring.cloud.stream.kafka.binder.transaction.producer.* properties.
Default null (no transactions)
Note that consumers on the output topic must be configured with isolation.level=read_committed to avoid receiving rolled-back records.
I have a Spring Cloud Stream application that receives events from RabbitMQ using the Rabbit Binder. My application can be summarized as this:
#Transactional
#StreamListener(MySink.SINK_NAME)
public void processEvents(Flux<Event> events) {
// Transform events and store them in MongoDB using
// spring-boot-data-mongodb-reactive
...
}
The problem is that it doesn't seem that #Transactional works with Spring Cloud Stream (or at least that's my impression) since if there's an exception when writing to MongoDB the event seems to have already been ack:ed to RabbitMQ and the operation is not retried.
Given that I want to achieve basically the same functionality as when using the #Transactional around a function with spring-amqp:
Do I have to manually ACK the messages to RabbitMQ when using Spring
Cloud Stream with the Rabbit Binder?
If so, how can I achieve this?
There are several issues here.
Transactions are not required for acknowledging messages
Reactor-based #StreamListener methods are invoked exactly once, just to set up the Flux so #Transactional on that method is meaningless - messages then flow through the flux so anything pertaining to individual messages has to be done within the context of the flux.
Spring Transactions are bound to the thread - Reactor is non-blocking; the message will be acked at the first handoff.
Yes, you would need to use manual acks; presumably on the result of the mongodb store operation. You would probably need to use Flux<Message<Event>> so you would have access to the channel and delivery tag headers.
I need to listen to a Rabbit Queue in the Flume Custom Source which I have developed.This requirement may seem awkward in Flume.But this is how its needed.
As I am using Spring AMQP to listen to the queue for simplicity,I am just not able to understand how to invoke the OnMessage() method within the Flume lifecycle Start() method,So that the messages can be posted onto the Flume channel.
I have looked at the Spring MessageListenerAdapter concept but I have not been able to find any example to implement the same.
onMessage() is a part of MessageListener pattern. It is some active component, which is initiated by the external system (from big height). And it works each time by that remote command, so you can't use it as a passive componet to be initiated by the user call.
Since you have "Flume lifecycle Start()" from other side and SimpleMessageListenerContainer has the same from its side, I'd say you have to correlate their lifecycles to work in tandem.
From here you should to provide for the SimpleMessageListenerContainer some inline MessageListener implementation, which invokes the desired method to "post onto the Flume channel".
HTH
UPDATE
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(connectionFactory);
....
container.setMessageListener(new MessageListener() {
public void onMessage(Message message) {
sendMessageToFlumeChannel(message);
}
});
Where the sendMessageToFlumeChannel is a method of the holding class.
Of course it can be any POJO instead of MessageListener implementation, but the main goal to delegate listener resul to some method.