I would like to consume messages from one kafka cluster and publish to another kafka cluster. Would like to know how to configure this using spring-kafka?
Simply configure the consumer and producer factories with different bootstrap.servers properties.
If you are using Spring Boot, see
https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-application-properties.html#spring.kafka.consumer.bootstrap-servers
and
https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-application-properties.html#spring.kafka.producer.bootstrap-servers
If you are creating your own factory #Beans, set the properties there.
https://docs.spring.io/spring-kafka/docs/current/reference/html/#connecting
you can use spring cloud stream kafka binders.
create two stream one for consume and one for producing.
for consumer
public interface InputStreamExample{
String INPUT = "consumer-in";
#Input(INPUT)
MessageChannel readFromKafka();
}
for producer
public interface ProducerStreamExample{
String OUTPUT = "produce-out";
#Output(OUTPUT)
MessageChannel produceToKafka();
}
for consuming message use:
#StreamListener(value = ItemStream.INPUT)
public void processMessage(){
/*
code goes here
*/
}
for producing
//here producerStreamExample is instance of ProducerStreamExample
producerStreamExample.produceToKafka().send(/*message goes here*/);
Now configure consumer and producer cluster using binder, you can use consumer-in for consumer cluster and produce-out for producing cluster.
properties file
spring.cloud.stream.binders.kafka-a.environment.spring.cloud.stream.kafka.binder.brokers:<consumer cluster>
#other properties for this binders
#bind kafka-a to consumer-in
spring.cloud.stream.bindings.consumer-in.binder=kafka-a #kafka-a binding to consumer-in
#similary other properties of consumer-in, like
spring.cloud.stream.bindings.consumer-in.destination=<topic>
spring.cloud.stream.bindings.consumer-in.group=<consumer group>
#now configure cluster to produce
spring.cloud.stream.binders.kafka-b.environment.spring.cloud.stream.kafka.binder.brokers:<cluster where to produce>
spring.cloud.stream.bindings.produce-out.binder=kafka-b #here kafka-b, binding to produce-out
#similary you can do other configuration like topic
spring.cloud.stream.bindings.produce-out.destination=<topic>
Refer to this for more configuration: https://cloud.spring.io/spring-cloud-stream-binder-kafka/spring-cloud-stream-binder-kafka.html
Related
If I have a topic called var test = "test"
and I create a bean:
#Bean(name = "test")
public Producer<Test> testProducer(PulsarClient pulsarClient){
return pulsarClient.newProducer()
.topic(test)
.create();
}
The topic is created I want to partition, how?
To create a topic with partitions you can either configure the following settings for the Pulsar Broker (if you are relying on automatic topic creation)
allowAutoTopicCreationType = partitioned
defaultNumPartitions = <N>
Alternatively you can use the Pulsar Admin and use createPartitionedTopic
https://github.com/apache/pulsar/blob/a165bda1d03e370f5efe1173134fb94e12b584b5/pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin/Topics.java#L473-L475
However once a topic has been created (I assume non-partitioned) then there is no way to make it partitioned. It has to be partitioned at topic creation time.
I have created a Kafka Consumer using Spring Cloud Stream and Spring Cloud Function for consuming messages in batch mode from a Kafka topic. Now, I want to send the error batches to a Dead Letter Queue for further debugging of the error.
I am handling retries inside my consumer method with Spring retry. But for non retry-able Exceptions I am looking to send the entire batch to a DLQ.
This is how my consumer looks like:
#Bean
public Consumer<List<GenericRecord>> consume() {
return (message) -> {
processMessage(message);
}
}
This is how the error handling Configuration looks like:
#Autowired
private DefaultErrorHandler errorHandler;
ListenerContainerCustomizer<AbstractMessageListenerContainer> c = new ListenerContainerCustomizer<AbstractMessageListenerContainer>() {
#Override
public void configure(AbstractMessageListenerContainer container, String destinationName, String group) {
container.setCommonErrorHandler(errorHandler);
}
}
The Error Handler is enabled with a DeadRecordPublishinRecoverer to send the failed messages to a DLQ:
#Bean
public DefaultErrorHandler errorHandler(KafkaOperations<String, Details> template) {
return new DefaultErrorHandler(new DeadLetterPublishingRecoverer(template,
(cr, e) -> new TopicPartition("error.topic.name", 0)),
new FixedBackOff(0, 0));
}
But this is not sending any message to the error.topic, And from the error logs I can see that it's trying to connect to localhost:9092 instead of the broker I have mentioned in spring.cloud.stream.kafka.binder.brokers.
How do I configure the DLQ provider to read the Kafka metadata from application.properties ?
Also is there a way to configure a Supplier function to create the DLQ provider?
You are probably using Boot's auto-configured KafkaTemplate.
Use spring.kafka.bootstrap-servers instead - the binder will use that if there is no spring.cloud.stream.kafka.binder.brokers; that way both the binder and template will connect to the same broker.
You have to throw a BatchListenerFailedException to indicate which record in the batch failed.
For our application we are writing the smae messages to 2 different Kafka Brokers with 2 different topics, now at the consumer part I have to consume both the topics and process only one topic that is enabled in spring boot property file.
How I can do it using spring boot and KafkaListner??
You can configure this simply by :
#KafkaListener(topics = { "${spring.kafka.topic1}", "${spring.kafka.topic2}" })
Now if you want to consume both but process one and discard the other you can use keys in the messages and create the key accordingly on both topics
Ref:
Java consumer API : ConsumerRecord#key()
I have a Producer producing messages in a RabbitMQ queue by using a direct exchange.
queue name: TEMP_QUEUE,
exchange name: TEMP_DIRECT_EXCHANGE
Producing to this queue is easy since on my producer application I use Spring AMQP which I am familiar with.
On my Consumer application, I need to use Spring cloud stream version 3.0+.
I want to avoid using legacy annotations like #EnableBinding, #StreamListener because they are about to be depracated.
Legacy code for my application would look like that :
#EnableBinding(Bindings.class)
public class TempConsumer {
#StreamListener(target = "TEMP_QUEUE")
public void consumeFromTempQueue(MyObject object) {
// do stuff with the object
}
}
public interface Bindings {
#Input("TEMP_QUEUE")
SubscribableChannel myInputBinding();
}
From their docs I have found out I can do something like that
#Bean
public Consumer<MyObject> consumeFromTempQueue() {
return obj -> {
// do stuff with the object
};
}
It is not clear to me how do I specify that this bean will consume from TEMP_QUEUE? Also what if I want to consume from multiple queues?
See Consuming from Existing Queues/Exchanges.
You can consume from multiple queues with
spring.cloud.stream.bindings.consumeFromTempQueue-in-0.destination=q1,q2,q3
spring.cloud.stream.bindings.consumer.multiplex=true
Without multiplex you'll get 3 bindings; with multiplex, you'll get 1 listener container listening to multiple queues.
You need to use the application.yml to bind your bean.
spring.cloud.stream:
function.definition: consumeFromTempQueue
You can use this configuration to configure source, process and sink as well. In your case you are just using a source.
You can read this post for more information.
I'm trying to achieve following thing:
Use Spring Cloud Stream 2.1.3.RELEASE with Kafka binder to send a message to an input channel and achieve publish subscribe behaviour where every consumer will be notified and be able to handle a message sent to a Kafka topic.
I understand that in Kafka if every consumer belongs to its own consumer group, will be able to read every message from a topic.
In my case spring creates an anonymous unique consumer group to every instance of my spring boot application running. The spring boot application has only one stream listener configured to listen to the input channel.
Test example case:
Configured an example Spring Cloud Stream app with an input channel which is bound to a Kafka topic.
Using a spring rest controller to send a message to the input channel expecting that message will be delivered to every spring boot application instance running.
In both applications on startup I can see that kafka partition is assigned properly.
Problem:
However, when I send a message using output().send() then spring doesn't even send a message to the Kafka topic configured, instead, in the same thread it triggers #StreamListener method of the same application instance.
During debugging I see that spring code has two handlers of the message. The StreamListenerMessageHandler and KafkaProducerMessageHandler.
Spring simply chains them and if first handler ends with success then it will not even go further. StreamListenerMessageHandler simply invokes my #StreamListener method in the same thread and message never reaches Kafka.
Question:
Is this by design, and in that case why is that? How can I achieve behaviour mentioned at the beginning of the post?
PS.
If I use KafkaTemplate and #KafkaListener method then it works as I want. Message is sent to Kafka topic and both application instances receive message and handles it in Kafka listener annotated method.
Code:
The stream listener method is configured the following way:
#SpringBootApplication
#EnableBinding(Processor.class)
#EnableTransactionManagement
public class ProcessorApplication {
private Logger logger =
LoggerFactory.getLogger(this.getClass().getName());
private PersonRepository repository;
public ProcessorApplication(PersonRepository repository) {
this.repository = repository;
}
public static void main(String[] args) {
SpringApplication.run(ProcessorApplication.class, args);
}
#Transactional
#StreamListener(Processor.INPUT)
#SendTo(Processor.OUTPUT)
public PersonEvent process(PersonEvent data) {
logger.info("Received event={}", data);
Person person = new Person();
person.setName(data.getName());
logger.info("Saving person={}", person);
Person savedPerson = repository.save(person);
PersonEvent event = new PersonEvent();
event.setName(savedPerson.getName());
event.setType("PersonSaved");
logger.info("Sent event={}", event);
return event;
}
}
Sending a message to the input channel:
#RestController()
#RequestMapping("/persons")
public class PersonController {
#Autowired
private Sink sink;
#PostMapping("/events")
public void createPerson(#RequestBody PersonEvent event) {
sink.input().send(MessageBuilder.withPayload(event).build());
}
}
Spring Cloud Stream config:
spring:
cloud.stream:
bindings:
output.destination: person-event-output
input.destination: person-event-input
sink.input().send
You are bypassing the binder altogether and sending it directly to the stream listener.
You need to send the message to kafka (to the person-event-input topic) and then each stream listener will receive the message from kafka.
You need to configure another output binding and send it there, not directly to the input channel.