I'm getting a bunch of deserialization failure before my Kafka Listener is hit. I was looking into the things Gary Russel built, but having issues getting it to work. All my stuff is configured via properties file.
spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.ErrorHandlingDeserializer2
spring.kafka.consumer.properties.spring.deserializer.value.delegate.class=io.confluent.kafka.serializers.KafkaAvroDeserializer
So if I add these, my understanding is it wraps an error in the headers of the consumer record? My ultimate goal is to have any deserialization exception hit some custom class I have so I can handle what I want to do with it. IE, forward to my dead letter handler which uploads failed data to s3.
I tried adding the errorhandler flag to the kafkalistener, but that also didn't do anything.
Updated Property Configuration
I've updated my configuration, it's still unclear to me if this is correct. It's not working, so I assume not.
None of the custom code is getting called
spring.kafka.consumer.properties.value.deserializer=org.springframework.kafka.support.serializer.ErrorHandlingDeserializer2
spring.kafka.consumer.properties.key.deserializer=org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
spring.kafka.consumer.properties.spring.deserializer.value.function=com.thing.cyclic.service.FailedFooProvider
spring.kafka.consumer.properties.spring.deserializer.key.delegate.class=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.properties.spring.deserializer.value.delegate.class=io.confluent.kafka.serializers.KafkaAvroDeserializer
spring.kafka.consumer.properties.spring.json.trusted.packages=*
spring.kafka.consumer.properties.value.subject.name.strategy=io.confluent.kafka.serializers.subject.TopicNameStrategy
spring.kafka.consumer.properties.specific.avro.reader=true
spring.kafka.consumer.properties.auto.register.schemas=false
spring.kafka.consumer.properties.isolation.level=read_committed
spring.kafka.listener.ack-mode=manual_immediate
BadFoo
public class BadFoo {
private final FailedDeserializationInfo failedDeserializationInfo;
public BadFoo(FailedDeserializationInfo failedDeserializationInfo) {
this.failedDeserializationInfo = failedDeserializationInfo;
}
public FailedDeserializationInfo getFailedDeserializationInfo() {
return this.failedDeserializationInfo;
}
}
FailedFooProvider
public class FailedFooProvider implements Function<FailedDeserializationInfo, String> {
#Override
public String apply(FailedDeserializationInfo info) {
System.out.println("");
return "";
}
}
See the documentation here and here.
Also take a look at the DeadLetterPublishingRecoverer code, which can used to publish the failed record to an other topic. You can model your code after that to obtain the header(s) containing the failed byte[].
https://github.com/spring-projects/spring-kafka/blob/fa5c35e9b15c4cecfc6ea2bbbf9e7745bc5d9f75/spring-kafka/src/main/java/org/springframework/kafka/listener/DeadLetterPublishingRecoverer.java#L169-L178
The recoverer is used in conjunction with a SeekToCurrentErrorHandler.
Configure the error handler as a #Bean and Spring Boot will automatically wire it into the container.
Related
I am trying to find a way to intercept Spring Cloud Stream 3.1.1 destination binding in order to modify some properties on the fly in my application. I have come to the following code snippet, but it seems it only works for a dynamic binding whereas in my case channels are defined specifically and no dynamic destination has been used, so I'm not sure what I am missing here.
#Bean
public NewDestinationBindingCallback<KafkaProducerProperties> bindingConfigurer() {
return ((channelName, channel, producerProperties, extendedProducerProperties) -> {
if ("foo".equals(channelName)) {
producerProperties.setUseNativeEncoding(producerUseNativeEncoding);
producerProperties.setPartitionCount(producerPartitionCount);
}
});
}
Spring boot Apache Camel-Java DSL app reads messages from Kafka topic.
#Component
public class KafkaTopicService extends RouteBilder {
public void configure(){
from("kafka:myTopic?brokers=localhost:9092")
.log("Message received from Kafka: ${body}")}
}
If I stop Kafka I get org.apache.kafka.common.errors.DisconnectException
I looked into onException(...class).handled(true) but Im not sure how to implement handling of the exception in my code. Can someone give me few implementation examples? What options are available? For example logging the message or reattempting to read message?
Documentation also mentions Quarkus. Do I need Quarkus to use onException()?
You can do something like (have not tried running it so please take care of any typos)
#Component
public class KafkaTopicService extends RouteBilder {
public void configure(){
onException(org.apache.kafka.common.errors.DisconnectException.class)
.log("Error connecting kafka");
from("kafka:myTopic?brokers=localhost:9092&bridgeErrorHandler=true")
.log("Message received from Kafka: ${body}")}
}
Please note that I have added bridgeErrorHandler=true. Normally exception handling happens after from. In most of the case using bridgeErrorHandler we can use onException function for those.
Also note that I have defined onException outside your route, so the exception handling logic which you add would be global and applicable to all routes wherever you encounter DisconnectException
Good Morning:
I'm new in Citrus Framework. Actually i work in a Test Case that consumes one soap webservice. I can send request message from a xml file and i need to store response message from server into another xml file for trazability and audit.
I try some options but still not working. Can you help me with posibles solutions to this requirement?
My test look like this:
public class DummyIT extends TestNGCitrusTestDesigner {
#Autowired
private WebServiceClient DummyClient;
#Test
#CitrusTest
public void dummyTest() {
soap()
.client(DummyClient)
.send()
.messageType(MessageType.XML)
.charset("UTF-8")
.contentType("text/xml")
.payload(new ClassPathResource("templates/DummyRequest.xml"));
soap()
.client(DummyClient)
.receive()
.schemaValidation(false);
}
I'm using Citrus Framework version 2.7.2.
Thanks for your help.
You can add a message tracing test listener to the Spring application context. This listener is called with all inbound and/or outbound messages. With a custom implementation you can write the message content as file to an external folder.
There is a default message listener implementation available that is a good starting point. See if this default tracing listener fits your requirements. Otherwise you would have to implement the listener logic on your own.
You can add the default listener to the application context as bean:
#Bean
public MessageTracingTestListener tracingTestListener() {
return new MessageTracingTestListener();
}
After that you should see .msgs files in target/citrus-logs/trace/messages folder containing all exchanged inbound and outbound messages.
Here is the default implementation: https://github.com/citrusframework/citrus/blob/master/modules/citrus-core/src/main/java/com/consol/citrus/report/MessageTracingTestListener.java
I working on an application which reads message from Azure service bus. This application was created using spring boot, Spring jms and Qpid jms client. I am able to read the message properly from Queue without any issues. PFB My code which I am using to read message.
#Service
public class QueueReceiver {
#JmsListener(destination = "testing")
public void onMessage(String message) {
if (null != message) {
System.out.println("Received message from Queue: " + message);
}
}}
Issue is we have different destinations for different environemnts, like testing for dev, testing-qa for qa and testing-prod for production, all these values are provided as azure.queueName in different application-(ENV).proerpties respectively. I want to pass these destinations dynamically to the destination in JmsListener Annotation. When i try using
#Value("${azure.queueName}")
private String dest;
and passing dest to annotation like #JmsListener(destination = dest)
I am getting The value for annotation attribute JmsListener.destination must be a constant expression Error. After googling with this Error i found that we cannot pass dynamic value to Annotation. Please help me how to resolve this issue or any other solution for this.
Use
destination="${azure.queueName}"
i.e. put the placeholder in the annotation directly.
You can use a dynamic name as defined in the application.properties file.For Example:
#JmsListener(destination = "${queue.name}")
Since you can't access any class variables here so this is the best option available.
I asked basically the same thing a few months ago with this post: How should a Spring JMS listener handle a message with an empty payload?, but all I got was a measly comment suggesting I "re-write my listener to do what I want". Valid statement, but unclear in my eyes as I'm still coming to grips with Spring-Boot. I've learned since then and want to re-ask this question more directly (as opposed to placing a bounty on the old one).
I set up an annotated bean class with #Configuration and #EnableJms and my container factory looks like:
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(mqConnectionFactory());
factory.setDestinationResolver(destinationResolver());
factory.setConcurrency("1");
factory.setErrorHandler(errorHandler());
factory.setSessionTransacted(true);
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
return factory;
}
And the listener looks like:
#JmsListener(id = "qID", destination = "qName")
public void processOrder(String message) {. . .}
As I understand it, once the annotated bean class gets ran through, the JMSListener basically kicks off (unless I set autoStartup to false), so I fail to understand where and when I have control over what or how the JmsListener handles things. From my perspective it "just runs". So if a queue has "\n" on it or just an empty string, the listener is going to throw an exception. Specifically org.springframework.messaging.converter.MessageConversionException: No converter found to convert to class java.lang.String. And this exception is thrown behind the scenes. I never get the chance to execute anything inside the listener
I looked into SimpleMessageConverter but didn't seem to see anything that would allow me to say something like setIgnoreStringPattern(). That obviously doesn't exist, but that's what I need. What am I missing? Is there a way to tell the JmsListener to ignore certain strings?
I took M. Deinum's suggestion (as it seemed quick and clean) and simply made the parameter type javax.jms.Message then converted the incoming message into a string. So my Listener now looks like
#JmsListener
public void processOrder(Message message) throws JMSException {
String convertedMessage = ((TextMessage) message).getText();
:
:
}
This may throw a JMSException, but I'm not too concerned with that as now when my implemented ErrorHandler class is called, I'll now know why and can do something more specific to handle a failed conversion. This does exactly what I need it to.
Edit: And in response to Jonh K's suggestion, the listener did not like having byte[] as a parameter. It basically wanted a converter to convert from byte array to string. Opted out of implementing my own custom converter.
#JmsListener(destination = "stompmessage")
public void receiveStomp(byte[] data, #Headers Map<Object, Object> allHeaders) {
System.out.println("Stomp message: "+ new String(data));
}
Version for spring in 2019-2020
You can add a custom message converter to the listener container factory and do whatever you want with the incoming message.