I have a problem with my kafka consumer, when consumer start reading from topic, message deserialization fails and block my consumer. Looks like it can't deserialize massage that has been changed. You can reproduce the problem, just produce message change it structure and produce it again on the same consumer. How I can handle this case and avoid a blocking of consumer?
Error message:
enter image description here
// initial message
class Message {
String name;
}
// Message after change
class Message {
List<String> names;
}
// When Exception throws, this code does not execute
#KafkaHandler
public void listenOnMessage(Message message) {
log.info("message: {}", message);
// ...
}
Update:
basically problem was in producer, it from some point in time started publish different messages structure, solution is simple, all I need to add is ErrorHandling on consumer side, retry logic and recovery. Basically discussion on reddit and repos with similar issue and solution you can find here https://www.reddit.com/r/apachekafka/comments/10dd0qg/how_to_handle_a_poison_pill_on_consumer_side/
Related
I'm trying to know when a message has been accepted (ack) or not (nack) using RabbitMQ and Spring Boot.
I want to send a message into a queue (via exchange) and check if the queue has been accepted the message. Actually I want to send to two different queues, but it is not important, I'm assuming if it works for one of them will work for the other too.
So I've tried something like this using CorrelationData:
public boolean sendMessage(...) {
CorrelationData cd = new CorrelationData();
this.rabbitTemplate.convertAndSend(exchange, routingKey, message, cd);
try {
return cd.getFuture().get(3, TimeUnit.SECONDS).isAck();
} catch (InterruptedException | ExecutionException | TimeoutException e ) {
e.printStackTrace();
return false;
}
}
The line cd.getFuture().get(3, TimeUnit.SECONDS).isAck() should get false is value has not been ack into the queue I think. But this is always true, even if routingKey doesn't exists.
So I'm assuming this piece of code is checking the message has been send into the exchange and exchange says "yes, I've recived the message, it has not been routed, but I've recived it".
So, I've looked for other ways into Rabbit/Spring documentation but I can't get the way.
And, explaining a little more, that I want is:
Into Spring Boot code I receive a message. This message has to been send to other queues/exchange, but can't be removed from the current queue (i.e. acked) until other two queues confirm the ack.
I have manual ack and as a little pseudo-code I have this:
#RabbitListener(queues = {queue})
public void receiveMessageFromDirect(Message message, Channel channel,
#Header(AmqpHeaders.DELIVERY_TAG) long tag){
boolean sendQueue1 = sendMessage(...);
boolean sendQueue2 = sendMessage(...);
if(sendQueue1 && sendQueue2){
//both messages has been readed; now I can ack this message
channel.basicAck(tag, false);
}else{
//nacked; I can't remove the message util both queue ack the message
channel.basicNack(tag,false,true);
}
I've tested this structure and, even if the queues don't exists, values sendQueue1 and sendQueue2 are always true.
The confirm is true; even for unroutable messages (I am not entirely sure why).
You need to enable returned messages (and check that it is null in the CorrelationData after the future completes - correlationData.getReturnedMessage()). If it's not null, the message wasn't routable to any queue.
You only get nacks if there is a bug in the broker, or if you are using a queue with x-max-length and overflow behavior reject-publish.
I'm writing an application with Spring Boot so to write to Kafka I do:
#Autowired
private KafkaTemplate<String, String> kafkaTemplate;
and then inside my method:
kafkaTemplate.send(topic, data)
But I feel like I'm just relying on this to work, how can I know if this has worked? If it's asynchronous, is it a good practice to return a 200 code and hoped it did work? I'm confused. If Kafka isn't available, won't this fail? Shouldn't I be prompted to catch an exception?
Along with what #mjuarez has mentioned you can try playing with two Kafka producer properties. One is ProducerConfig.ACKS_CONFIG, which lets you set the level of acknowledgement that you think is safe for your use case. This knob has three possible values. From Kafka doc
acks=0: Producer doesn't care about acknowledgement from server, and considers it as sent.
acks=1: This will mean the leader will write the record to its local log but will respond without awaiting full acknowledgement from all followers.
acks=all: This means the leader will wait for the full set of in-sync replicas to acknowledge the record.
The other property is ProducerConfig.RETRIES_CONFIG. Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error.
Yes, if Kafka is not available, that .send() call will fail, but if you send it async, no one will be notified. You can specify a callback that you want to be executed when the future finally finishes. Full interface spec here: https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/producer/Callback.html
From the official Kafka javadoc here: https://kafka.apache.org/20/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
Fully non-blocking usage can make use of the Callback parameter to
provide a callback that will be invoked when the request is complete.
ProducerRecord<byte[],byte[]> record = new ProducerRecord<byte[],byte[]>("the-topic", key, value);
producer.send(myRecord,
new Callback() {
public void onCompletion(RecordMetadata metadata, Exception e) {
if(e != null) {
e.printStackTrace();
} else {
System.out.println("The offset of the record we just sent is: " + metadata.offset());
}
}
});
you can use below command while sending messages to kafka:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic-name
while above command is running you should run your code and if sending messages being successful then the message must be printed on the console.
Furthermore, likewise any other connection to any resources if the connection could not be established, then doing any kinds of operations would result some exception raises.
Kafka allows for asynchronous message sending through below methods on Producer (KafkaProducer) class:
public java.util.concurrent.Future<RecordMetadata> send(ProducerRecord<K,V> record)
public java.util.concurrent.Future<RecordMetadata> send(ProducerRecord<K,V> record, Callback callback)
Successes can be handled through
1) the Future<RecordMetaData> object or
2) onCompletion method invoked by the callback. Full method signature and usage of onCompletion is as below (taken from kafka docs)
`
ProducerRecord<byte[],byte[]> record = new ProducerRecord<byte[],byte[]>("the-topic", key, value);
producer.send(record,
new Callback() {
public void onCompletion(RecordMetadata metadata, Exception e) {
if(e != null)
e.printStackTrace();
System.out.println("The offset of the record we just sent is: " + metadata.offset());
}
});
While failure needs to be handled through the Exception e passed to the onCompletion method
Fine every thing looks good so far.
But if I am getting it right, any reasonable information that can be obtained from exception or e object is stacktrace and exception message. What I mean to point out here is, e does not contain any information of the actual record sent. Or in other words, it does not contain a reference to the actual record that was sent to kafka broker. So what useful processing or handling can be done by the producer if the record was not sent successfully. Really not much.
Why I say this is - ideally I would like to make a log of the failed message some where and then try to resend it. But with the little information (e) provided by framework, i feel this is not possible.
Can someone point out if I am right or wrong?
You could easily create a callback that receives the producerRecord as a constructor argument. So upon onCompletion with an exception, you can have complete knowledge of the producer record, and even try to send it again.
I dealt with the same issue. Created a callback that gets both producerRecord, and a callback handler that uses an executor service to send the record again. So eventually, I can tolerate any number of failures (e.g. network issues or kafka is down), and recover from it.
I encountered an interresting and I think very common synchronization problem in my test code.
This is the test (its a functional test that connects from the outside to the system), i run it via TestNG.
#Test
public void operationalClientConnected_sendGetUserSessionRequest_clientShallReceiveGetUserSessionResponse() {
// GIVEN
OperationalClientSimulator client = operationalClientHasEstablishedWebSocketConnection("ClientXY");
// WHEN
GetUserSessionRequest request = PojoRequestBuilder.newRequest(GetUserSessionRequest.class).build();
client.sendRequest(request);
// THEN
assertThatClientReceivesResponse(client, GetUserSessionResponse.class, request.getCorrelationId(), request.getRequestId());
}
Basically i send a single request and wait for the correct response, this is what i want to verify in this test.
Behind the assertThatClientReceivesResponse there is a hamcrest matcher that looks like this:
#Override
protected boolean matchesSafely(final OperationalClientSimulator client) {
Object awaitedMessage = client.awaitMessage(
new Verification<Object>() {
#Override
public VerificationResult verify(final Object actual) {
VerificationResult result = new VerificationResult();
if (!_expectedResponseClass.isInstance(actual)) {
result.addMismatch("not of expected type", actual, _expectedResponseClass.getSimpleName());
}
// check more details of message ..
return result;
}
}, _expectedTimeout);
boolean matches = awaitedMessage != null;
if (matches) {
_messageCaptor.setActualMessage((T) awaitedMessage);
}
return matches;
}
Now to the interresting part, the synchronization in the OperationalClientSimulator class.
Two methods are of interrest:
awaitMessage which blocks until either a message that matches the given Verification is received or the timeout expired
onMessage received method which is called for each message that is received (over a websocket connection)
Basically what I want to achive is having the test thread block on the awaitMessage method until either the correct message is received (via onMessage) or the specified timeout elapsed.
public Object awaitMessage(final Verification<Object> verification, final long timeoutMillis) {
// howto sync?
return awaitedMessage; // or null
}
#Override
public void onMessage(final String message) {
LOG.info("#Client {} <== received a message on websocket - {}", name, message);
// howto sync?
}
About the test:
The test thread will almost always be faster and therefor has to wait until the response is received via the awaitMessage method
There can be very rare cases when the expected message is received before the test thread is checking for it (this basically means i have to save every received message)
In this specific test case there are only a handfull of messages that are received (some heartbeat messages, the actual response and a notification), but in other cases there can be hundreds of messages which in need to inspect to find the expected message(s)
I was thinking about different solutions for synchronizing here:
The simplest of course would be the sync with the synchronized keyword but I think there are neater ways to do this
The onMessage received method could simply write into a blocking queue and the test thread can consume from it but here I dont know how to measure the timeout.. can I use a CountdownLatch?
Maybe I can do a non blocking solution where the producer (onMessage) writes into an Array and the consumer reads until it reaches an index that is published by the producer (like the LMAX Disruptor)
I know this is test code and performance is not really an issue here, I am just thinking how to solve this in a "nice" way.. you know.. because its christmas :-)
So the actual question here is, how do i "safely" wait for the message which i expect in my test with a timeout? Safely here means that i never miss a message or lose a message because of concurrency issues and that I also need to check if the expected message was already received.
How should I synchronize between the test runner thread and the thread that calls the onMessage method in the OperationalClientSimulator when a message is received on the websocket connection.
I am using ActiveMQ to send the message.
So when I sent a message, the message comes to receive message. On successful insertion, it is acknowledged.
But I have code after acknowledgement, which can throw NullPointerException.
So to produce that exception intentionally, I have thrown NullPointerException.
So when it does that:
Message is not dequeued and the same message comes again to the onMessage function.
My code is:
public void onMessage(Message message) {
String msg = null;
try
{
msg = receiveMessage(message);
// Other code to insert message in db
message.acknowledge();
if(true)
{
throw new NullPointerException("npe"));
}
** // Other code which might produce a null pointer exception **
}
catch(Exception ex)
{
}
}
Why is the message again coming to onMessage() function as I have acknowledge() it also.
Since I have already inserted the message in db.
Doesn't the message inside queue will be removed on acknowledge()?
How I can achieve this?
You use AUTO acknowledge mode with message listners, then by specification, a message is redelivered if the message listeners fails to return successfully (for instance if there is an exception thrown).
In your case, you are trying to manually acknowledge the message, but that is not possible using a session created with createSession(false, Session.AUTO_ACKNOWLEDGE).
Your code would have worked with Session.CLIENT_ACKNOWLEDGE.
Otherwise, you want to catch the exceptions inside the onMessage method, while using AUTO_ACKNOWLEDGE.
To get a more fine grained controll over your messages, please consider using transacted sessions and use session.commit(); to confirm a message has been read.
Have you checked that you are not using transacted sessions?. When using transacted sessions,the acknowledge mode is ignored, so:
Your message.acknowledge() would effectively be a no-op
Your uncaught exception would be triggering a "session rollback" when escaping your message listener, forcing redelivery of the message.
NOTE: Your published code has a catch (Exception ex) { }, so I don't know exactly how your exception escapes outside.
You can create a separate method for processing the message, by which I mean that in the onMessage() function write code for only insertion of that message into the database.
And create a separate function for the processing of that message.
So that if you get any error during processing, the message will not come to onMessage() again.
When you use a transacted JMS acknowledge mode, your message will be received by JMS-listener several times (in AMQ by default it is approximately eight) till be processed without exception or will be moved by JMS-container to DQL-queue. See Message Redelivery and DLQ Handling for details.
Managing transactions depends on the framework used by you. I prefer to use Spring Framework, so my Spring XML configuration is looks like:
<jms:listener-container container-type="default"
connection-factory="calendarConnectionFactory"
acknowledge="transacted"
destination-type="queue"
cache="consumer"
concurrency="1-5">
<jms:listener destination="${jms.calendar.destination}" ref="calendarListener"/>
</jms:listener-container>
And the Java code of my message listener is
#Override
#Transactional(propagation = Propagation.REQUIRED,
noRollbackFor =
{ClassCastException.class, IllegalArgumentException.class})
public void onMessage(Message message) {
....
}
So I can manage what exceptions will rollback the transaction or not.