I am trying to implement exponential backoff for consumer failures. To that end I have three queues with DLX thus: RETRY -> MAIN -> FAILED.
Anything rejected from MAIN goes FAILED, and anything added to RETRY goes into MAIN after a per-message TTL. The consumer receives from MAIN.
I've implemented an ErrorHandler and set it on the SimpleRabbitListenerContainerFactory. This handler either computes a new TTL and sends the message to the RETRY queue, or throws AmqpRejectAndDontRequeueException if that's not possible or retries are exceeded in order to DLX it to FAILED. The problem is, I cannot work out how to get rid of the original message.
As far as I can see I have to ack it, but the Channel is not available in the error handler, and there are no other exceptions to throw that would trigger an ack.
If instead I remove the MAIN -> FAILED DLX and switch to manually adding messages to FAILED, then if that doesn't work I've lost the message.
#Override
public void handleError(Throwable t) {
log.warn("Execution of Rabbit message listener failed.", t);
try {
queueForExponentialRetry(((ListenerExecutionFailedException) t).getFailedMessage());
// what to do here?
} catch (RuntimeException ex) {
t.addSuppressed(ex);
log.error("Not requeueing after failure", t);
throw new AmqpRejectAndDontRequeueException(t);
}
// or here?
}
I think I immediately found the answer. Missed it before because I was throwing from the the wrong place.
#Override
public void handleError(Throwable t) {
log.warn("Execution of Rabbit message listener failed.", t);
try {
queueForExponentialRetry(((ListenerExecutionFailedException) t).getFailedMessage());
} catch (RuntimeException ex) {
t.addSuppressed(ex);
log.error("Not requeueing after failure", t);
throw new AmqpRejectAndDontRequeueException(t);
}
throw new ImmediateAcknowledgeAmqpException("Queued for retry");
}
ImmediateAcknowledgeAmqpException
Special exception for listener implementations that want to signal that the current batch of messages should be acknowledged immediately (i.e. as soon as possible) without rollback, and without consuming any more messages within the current transaction.
This should be safe as I'm not using batches or transactions, only publisher returns.
Side note: I should also be aware that exponential backoff isn't going to actually work properly:
While consumers never see expired messages, only when expired messages reach the head of a queue will they actually be discarded (or dead-lettered). When setting a per-queue TTL this is not a problem, since expired messages are always at the head of the queue. When setting per-message TTL however, expired messages can queue up behind non-expired ones until the latter are consumed or expired.
Related
try{
RecordMetadata d = producer.send(message).get()
if(d.hasoffset()){
//only way to ensure kafka was sent??
}}
catch (Exception e){
//or does this alone guarentee the message was sent
}
does the catch exception alone ensure the message was sent or is some if statement check required
You should set acks=1 or acks=all to ensure a broker gets the messages.
After that, then if .get() does not throw an exception, then the broker will have the record.
You'll also want to ensure the min.insync.replicas on the topics are at least 1.
Kafka allows for asynchronous message sending through below methods on Producer (KafkaProducer) class:
public java.util.concurrent.Future<RecordMetadata> send(ProducerRecord<K,V> record)
public java.util.concurrent.Future<RecordMetadata> send(ProducerRecord<K,V> record, Callback callback)
Successes can be handled through
1) the Future<RecordMetaData> object or
2) onCompletion method invoked by the callback. Full method signature and usage of onCompletion is as below (taken from kafka docs)
`
ProducerRecord<byte[],byte[]> record = new ProducerRecord<byte[],byte[]>("the-topic", key, value);
producer.send(record,
new Callback() {
public void onCompletion(RecordMetadata metadata, Exception e) {
if(e != null)
e.printStackTrace();
System.out.println("The offset of the record we just sent is: " + metadata.offset());
}
});
While failure needs to be handled through the Exception e passed to the onCompletion method
Fine every thing looks good so far.
But if I am getting it right, any reasonable information that can be obtained from exception or e object is stacktrace and exception message. What I mean to point out here is, e does not contain any information of the actual record sent. Or in other words, it does not contain a reference to the actual record that was sent to kafka broker. So what useful processing or handling can be done by the producer if the record was not sent successfully. Really not much.
Why I say this is - ideally I would like to make a log of the failed message some where and then try to resend it. But with the little information (e) provided by framework, i feel this is not possible.
Can someone point out if I am right or wrong?
You could easily create a callback that receives the producerRecord as a constructor argument. So upon onCompletion with an exception, you can have complete knowledge of the producer record, and even try to send it again.
I dealt with the same issue. Created a callback that gets both producerRecord, and a callback handler that uses an executor service to send the record again. So eventually, I can tolerate any number of failures (e.g. network issues or kafka is down), and recover from it.
Could you please clarify the following problem. Is it possible to return message to queue in case Message Driven Bean could not process a message. My code likes :
public void onMessage(Message message) {
try {
doSomethingWithMessage(message)
} catch (QueueListenerUtilException e) {
LOG.error("Could not process given message try to rollback transaction");
mdc.setRollbackOnly();
throw e;
}
LOG.debug("Sending message has been started");
this.simCntrlUtil.writeToQueue(answer, message, msgProp);
LOG.info(" onMessage has been completed");
}
I suppose that if bean throws RuntimeException the transaction will be rollbacked and after timeout will be delivered again. In server log I see message
"Resources rolled back due to setRollbackOnly() being called"
But message is not delivered again after timeout. I think that corresponded options are set in WebSphere configuration.
this
Automatically stop endpoints on repeated message failure
Enable
Sequential failed message threshold
100
Delay between failing message retries
10000
milliseconds
The transaction is managed by container (I did not change dedault initialization)
Could you please help me to understand : why I don't see message again?
Thank you in advance
I have the following listener method:
#Override
public void onMessage(Message message, Channel channel) {
try {
// do something bad :)
} catch (Exception e){
try {
long dt = null != message.getMessageProperties()
? message.getMessageProperties().getDeliveryTag()
: 0;
channel.basicReject(dt, true);
} catch(IOException io) {
logger.error("IO-COMMON", io);
}
}
}
The issue is basic reject doesn't work, I don't know why. How to reject it gracefully? I think that if I reject a message, it should be requeued and reside is sth like cache, before going to next worker. But in fact this message just seems to be lost.
You need to set the acknowledgemode to MANUAL if you are doing your own acks. I am not sure why it's not working for you; DEBUG/TRACE logging might help.
You should consider letting the container handle the acks - use acknowledgemode=AUTO; the container will normally requeue the message for any exception thrown or ack it if the listener returns normally.
You can set defaultRequeueRejected to false (it is true by default) and the message will be discarded (or routed to a DLX/DLQ).
You can also throw an AmqpRejectAndDontRequeueException to override the default mechanism of requeuing failed messages.
If the ack mode is NONE - there are no acks and RabbitMQ automatically acks the message as soon as it's sent.
I am using ActiveMQ to send the message.
So when I sent a message, the message comes to receive message. On successful insertion, it is acknowledged.
But I have code after acknowledgement, which can throw NullPointerException.
So to produce that exception intentionally, I have thrown NullPointerException.
So when it does that:
Message is not dequeued and the same message comes again to the onMessage function.
My code is:
public void onMessage(Message message) {
String msg = null;
try
{
msg = receiveMessage(message);
// Other code to insert message in db
message.acknowledge();
if(true)
{
throw new NullPointerException("npe"));
}
** // Other code which might produce a null pointer exception **
}
catch(Exception ex)
{
}
}
Why is the message again coming to onMessage() function as I have acknowledge() it also.
Since I have already inserted the message in db.
Doesn't the message inside queue will be removed on acknowledge()?
How I can achieve this?
You use AUTO acknowledge mode with message listners, then by specification, a message is redelivered if the message listeners fails to return successfully (for instance if there is an exception thrown).
In your case, you are trying to manually acknowledge the message, but that is not possible using a session created with createSession(false, Session.AUTO_ACKNOWLEDGE).
Your code would have worked with Session.CLIENT_ACKNOWLEDGE.
Otherwise, you want to catch the exceptions inside the onMessage method, while using AUTO_ACKNOWLEDGE.
To get a more fine grained controll over your messages, please consider using transacted sessions and use session.commit(); to confirm a message has been read.
Have you checked that you are not using transacted sessions?. When using transacted sessions,the acknowledge mode is ignored, so:
Your message.acknowledge() would effectively be a no-op
Your uncaught exception would be triggering a "session rollback" when escaping your message listener, forcing redelivery of the message.
NOTE: Your published code has a catch (Exception ex) { }, so I don't know exactly how your exception escapes outside.
You can create a separate method for processing the message, by which I mean that in the onMessage() function write code for only insertion of that message into the database.
And create a separate function for the processing of that message.
So that if you get any error during processing, the message will not come to onMessage() again.
When you use a transacted JMS acknowledge mode, your message will be received by JMS-listener several times (in AMQ by default it is approximately eight) till be processed without exception or will be moved by JMS-container to DQL-queue. See Message Redelivery and DLQ Handling for details.
Managing transactions depends on the framework used by you. I prefer to use Spring Framework, so my Spring XML configuration is looks like:
<jms:listener-container container-type="default"
connection-factory="calendarConnectionFactory"
acknowledge="transacted"
destination-type="queue"
cache="consumer"
concurrency="1-5">
<jms:listener destination="${jms.calendar.destination}" ref="calendarListener"/>
</jms:listener-container>
And the Java code of my message listener is
#Override
#Transactional(propagation = Propagation.REQUIRED,
noRollbackFor =
{ClassCastException.class, IllegalArgumentException.class})
public void onMessage(Message message) {
....
}
So I can manage what exceptions will rollback the transaction or not.