I currently switched to version 0.24.0-beta of the google-cloud-pubsub library and can't find a way to modify acknowledge deadline of the message. In the previous version there was a modifyAckDeadline(), which allowed me to do it. Is it possible in the new version?
To get an idea why I'm trying to it: What I want to achieve is a kind of retry mechanism where messages that failed to process are redelivered in 10 minutes, and acked only after X such unsuccessful retries.
For purposes of keeping the lease on a message because it will take longer to process, the new version of the subscriber library calls modifyAckDeadline in the background for you. As long the message is outstanding, i.e., you haven't called ack() or nack() on the AckReplyConsumer that your MessageReceiver receives with the message, the library itself will call modifyAckDeadline on the message.
Your goal is a little different: you want to use modifyAckDeadline to best-effort prevent message redelivery for a time. If the time you want to wait is always going to be 10 minutes, then the best thing to do is to call setMaxAckExtensionPeriod() with a duration of 10 minutes on the Subscriber.Builder when creating your subscriber. Then, when you fail to process a message, just don't ack or nack the message. The client library will continue to modifyAckDeadline for ten minutes, after which the message can be redelivered.
Related
Regard new feature of Kafka aimed for negative acknowledgement and now supported by Spring-Kafka, according to /spring-kafka/docs/2.4.4.RELEASE/
"...
Starting with version 2.3, the Acknowledgment interface has two additional methods nack(long sleep) and nack(int index, long sleep). The first one is used with a record listener, the second with a batch listener. Calling the wrong method for your listener type will throw an IllegalStateException.
...
With a record listener, when nack() is called, any pending offsets are committed, the remaing records from the last poll are discarded, and seeks are performed on their partitions so that the failed record and unprocessed records are redelivered on the next poll(). The consumer thread can be paused before redelivery, by setting the sleep argument. This is similar functionality to throwing an exception when the container is configured with a SeekToCurrentErrorHandler.
"
Well, if some error happens on consumer side, say fail to save on database, let's say the consumer doesn't acknowledgment.acknowledge(), as far as I understand the message is still on poll and it will be read/consumed again. I guess someone can say that with nack(..., some time) the consumer can sleep giving the chance to read/consume again a bit later and don't face error. If keep listening the topic isn't an issue, my straight question is:
is there any further point on using nack instead of simply not acknowledge?
As far as I can see the message will keep in pool for the time longer than the nack sleep anywhay. So, by the way, if the consumer keeps trying get the message and save the message it will successed assuming the issue is fixed in less than sleep time.
A surrounding point or advantage would be that somehow the producer get notified that nack is used. If so, I could find some worth on it in some specific scenarios. Let's say while using Log Compation (interested only on last message status) or Kafka as a long-term storage service (future releases will provide this I guess - KIP 405)
Regard more general exceptions I tend to follow approaches like configure a SeekToCurrentErrorHandler and throw the exception
nack is simply an alternative to using a SeekToCurrentErrorHandler - it was added before we made the SeekToCurrentErrorHandler the default error handler (previously, the default simply logged the error).
The STCEH is more sophisticated in that you can configure the number of retries, configure a recoverer called after retries are exhausted (e.g. DeadLetterPublishingRecoverer).
nack is different to "not acknowledge"; with the latter, if you don't throw an exception, you'll get the next record from the poll (or the next poll if this was the last record); you will not get a redelivery of a not-acked record unless you either use nack or a STCEH and throw an exception.
Sometimes due to some external problems, I need to requeue a message by basic.reject with requeue = true.
But I don't need to consume it immediately because it will possibly fail again in a short time. If I continuously requeue it, this may result in infinite loop and requeue.
So I need to consume it later, say one minute later,
And I need to know how many times the messages has been requeue so that I can stop requeue it but only reject it to declare it fails to consume.
PS: I am using Java client.
There are multiple solutions to point 1.
First one is the one chosen by Celery (a Python producer/consumer library that can use RabbitMQ as broker). Inside your message, add a timestamp at which the task should be executed. When your consumer gets the message, do not ack it and check its timestamp. As soon as the timestamp is reached, the worker can execute the task. (Note that the worker can continue working on other tasks instead of waiting)
This technique has some drawbacks. You have to increase the QoS per channel to an arbitrary value. And if your worker is already working on a long running task, the delayed task wont be executed until the first task has finished.
A second technique is RabbitMQ-only and is much more elegant. It takes advantage of dead-letter exchanges and Messages TTL. You create a new queue which isn't consumed by anybody. This queue has a dead-letter exchange that will forward the messages to the consumer queue. When you want to defer a message, ack it (or reject it without requeue) from the consumer queue and copy the message into the dead-lettered queue with a TTL equal to the delay you want (say one minute later). At (roughly) the end of TTL, the defered message will magically land in the consumer queue again, ready to be consumed. RabbitMQ team has also made the Delayed Message Plugin (this plugin is marked as experimental yet fairly stable and potential suitable for production use as long as the user is aware of its limitations and has serious limitations in term of scalability and reliability in case of failover, so you might decide whether you really want to use it in production, or if you prefer to stick to the manual way, limited to one TTL per queue).
Point 2. just requires putting a counter in your message and handling this inside your app. You can choose to put this counter in a header or directly in the body.
I'm using Camel for a while and I'm a huge admirer of its simplicity.
The use case
Given this simple route:
from("mock:some-route")
// split 1
.split().method("splitterBean", "split")
// now we have an ArrayList of n messages (let's say 10)
.to(ExchangePattern.InOut, "jms:some-thing");
If we assume that we have 10 messages after the split(), this route will immediately send 10 messages to the "to" endpoint. So jms:some-thing will receive all 10 messages at once.
The problem
--> Please note that the "out" endpoint is inOut, so we have timeouts in place when the receiver must acknowledge the message.
The application on the receiving end of jms:some-thing has to do quite some work for each message. As all 10 messages were written at the same time, the same timeout applies for all of them.
So we increased that timeout.
But one day, we will have 1000 messages and the timeout will again be to low.
What i want to archieve
I want to implement a pattern where I'll send only 1 message at once after the split, then sending the next after 1 message is acknowledged by the receiving system.
So instead of sending the 10 messages at once, I want
Send 1 message
Wait for the acknowledgment of that message
Send the next
Wait again
And so on..
How to implement such behavior?
I looked at the documentation, but none of the EIP components seem to fulfill that need?
Thanks for any input
You can have an intermediate seda queue with only one thread.
from("mock:some-route")
.split().method("splitterBean", "split")
.to("seda:your-seda-queue?waitForTaskToComplete=Always&timeout=0");
from("seda:your-seda-queue?waitForTaskToComplete=Always&timeout=0")
.to(ExchangePattern.InOut, "jms:some-thing");
By default, a seda queue will have a single consuming thread, and will block the calling thread until a consumer becomes available. More on seda details here
Saying that, your sending to a jms topic, which is really what you should be using to queue up your requests instead of a seda queue. You should look into implementing this logic asynchronously, and waiting on a reply topic rather than using a timeout.
I have a system that reads messages from a IBM MQ. From this queue we sometimes get corrupt data. If a message has failed the next time it is read the listener will wait 5 minutes before proceeding. When a message has failed 3 times it is put on a Backoff-queue. The JMSContainerFacotry is set up to run max 10 in parallell. This means if we get a big batch of corrupt messages these messages take 10+ minutes each to reach the Backoff-queue (which is ok), but they end up consuming all the threads so other messages must wait for all the corrupt to reach the Backoff-queue before being processed. I want to remove the wait() from the listener so that this doesnt block threads, but that failed messages still wait 5 minutes before popping back up.
I know that with ActiveMQ you can tell the queue to wait 5 minutes before popping the same message again, with IBM MQ I dont have that possibility. Is it possible to put failed messages to the back off the queue? Now failed messages seems to pop in front of new ones. Or set the jmspriority on failed messages lower. So that if there is messages that have yet to fail will get popped from the queue before those that have failed?
My listener is about so:
#JMSListener()
public void listen(Object message) {
TextMessage textMessage = (TextMessage) message;
if (textMessage.getIntProperty("JMSXDeliveryCount") > 1) {
//pause for 5 minutes
}
//buisness logic
}
The in-queue is a IBM MQ. AS is Jboss 6.
Not sure why you're pausing for 5 minutes if you get a poison message. If you know it's bad why aren't you either throwing it away or passing it to a backout queue for further investigation? See JMS poison message handling:
http://www.ibm.com/support/knowledgecenter/SSFKSJ_9.0.0/com.ibm.mq.dev.doc/q032280_.htm
More concerning, why are you getting so many 'bad' messages?
I know that with ActiveMQ you can tell the queue to wait 5 minutes
before popping the same message again, with IBM MQ I dont have that
possibility.
The option of retry interval between attempts need to be set on the MDB listener side, since it is acting as client. But not on WMQ side.
Is it possible to put failed messages to the back off the queue?
You wanted to place the message back on the queue or place the poisonous messages at the end of the same queue ? we can't move poison messages to the end of the same queue
There are only three options to handle poisonous messages:
Place the message back on the queue
re-route the message either to BOQ or DLQ
Purge or discard it permanently.
I am using ActiveMQ version 5.7.x,
I am having one ActiveMQ queue to which the listener listens.
Queue has a ConnectionFactory whose redeliveryPolicy is set to 3, intialRedeliveryDelay set to 5000.
Queue have some good messages and bad messages. While listening to such queue, when bad messages come, they are tried 3 times with the wait time of 5000 millis, but then the good messages are blocked for that much time span.
What I want is, during the wait time of 5000 millis for each retry, the processing of good messages should continue and should not wait for bad message processing.
For this I tried 1 attribute of connectionFactory, i.e. nonBlockingRedelivery set to true.
But nonBlockingRedelivery is not working.
Is there any other way to do this?
You can always have a retransmission queue for failing messages.
That is, receive messages from a main queue (no redelivery) and if you get an exception, put the message on a redelivery queue.
Let your application listen on both queues and do the same logic to both messages. It should simply be two message listeners invoking the same method. One with redelivery and one without, but with slightly different error handling.
Please look at the description of that attribute in ActiveMQConnectionFactory,
It says what I exactly what I wanted..
"When true a MessageConsumer will not stop Message delivery before re-delivering Messages from a rolled back transaction. This implies that message order will not be preserved and also will result in the TransactedIndividualAck option to be enabled"
But same is not working..!!!
Can you please look into it please?