Using Spring-boot and Spring-starter-amqp for my messaging app. My Problem scenario is this :
First Problem : I want to requeue my message only for 5 times and then message should be comeout from queue if business exception occurs(here I am focusing only on business exception).
Second Problem : If first use case is possible, can we somehow increase time to requeue based on the attempts. Suppose in the first attempt business exception occurs it should requeue immediately but in the second attempt it should requeue after 2 mins then 4 mins then 6 mins then 8 mins then 10 mins like time will increase based on the attempt.
Thanks
For the simple case (fixed delay between attempts), you can configure the queue to send rejected messages a dead letter exchange, and throw AmqpRejectAndDontRequeueException.
Bind a queue to the dead letter exchange, and set a time to live (ttl) on that queue and also configure it with a dead letter exchange to which the original queue is bound. The message will be requeued after the TTL expires.
You would need to examine the x-death header to determine how many times around the loop the message has traversed. After your retries are exhausted, throw an ImmediateAcknowledgeAmqpException to discard the message.
For a variable delay, you have to republish the message yourself, to a delayed-message exchange (there's a broker plugin for that), with the delay increasing for each retry.
Also see this answer.
Related
I have two lambda instance running at the same time and these two instances will do a short poll to the same FIFO queue with only a few seconds apart.
The first instance will receive the first 10 messages and the second instance will receive 0 message even though there are a total of 15 messages in the queue.
Why couldn't the second instance get the remaining 5 messages from the queue? Is this the expected behaviour and how can I overcome it?
Your 15 messages (most likely) all belong to the same Message Group ID. Therefore, the remaining 5 will not become available to your consumers until the first 10 are successfully processed and deleted. For FIFO Queues, this is the expected behaviour to preserve the order of messages (cheers #Michael-sqlbot for pointing in the right direction with this answer as per comments below).
Use long polling for Standard Queues. Short polling doesn't check every SQS server, therefore, it has the potential to not get all results. Long polling does check all SQS servers and will therefore get all results.
Sometimes due to some external problems, I need to requeue a message by basic.reject with requeue = true.
But I don't need to consume it immediately because it will possibly fail again in a short time. If I continuously requeue it, this may result in infinite loop and requeue.
So I need to consume it later, say one minute later,
And I need to know how many times the messages has been requeue so that I can stop requeue it but only reject it to declare it fails to consume.
PS: I am using Java client.
There are multiple solutions to point 1.
First one is the one chosen by Celery (a Python producer/consumer library that can use RabbitMQ as broker). Inside your message, add a timestamp at which the task should be executed. When your consumer gets the message, do not ack it and check its timestamp. As soon as the timestamp is reached, the worker can execute the task. (Note that the worker can continue working on other tasks instead of waiting)
This technique has some drawbacks. You have to increase the QoS per channel to an arbitrary value. And if your worker is already working on a long running task, the delayed task wont be executed until the first task has finished.
A second technique is RabbitMQ-only and is much more elegant. It takes advantage of dead-letter exchanges and Messages TTL. You create a new queue which isn't consumed by anybody. This queue has a dead-letter exchange that will forward the messages to the consumer queue. When you want to defer a message, ack it (or reject it without requeue) from the consumer queue and copy the message into the dead-lettered queue with a TTL equal to the delay you want (say one minute later). At (roughly) the end of TTL, the defered message will magically land in the consumer queue again, ready to be consumed. RabbitMQ team has also made the Delayed Message Plugin (this plugin is marked as experimental yet fairly stable and potential suitable for production use as long as the user is aware of its limitations and has serious limitations in term of scalability and reliability in case of failover, so you might decide whether you really want to use it in production, or if you prefer to stick to the manual way, limited to one TTL per queue).
Point 2. just requires putting a counter in your message and handling this inside your app. You can choose to put this counter in a header or directly in the body.
I'm using Camel for a while and I'm a huge admirer of its simplicity.
The use case
Given this simple route:
from("mock:some-route")
// split 1
.split().method("splitterBean", "split")
// now we have an ArrayList of n messages (let's say 10)
.to(ExchangePattern.InOut, "jms:some-thing");
If we assume that we have 10 messages after the split(), this route will immediately send 10 messages to the "to" endpoint. So jms:some-thing will receive all 10 messages at once.
The problem
--> Please note that the "out" endpoint is inOut, so we have timeouts in place when the receiver must acknowledge the message.
The application on the receiving end of jms:some-thing has to do quite some work for each message. As all 10 messages were written at the same time, the same timeout applies for all of them.
So we increased that timeout.
But one day, we will have 1000 messages and the timeout will again be to low.
What i want to archieve
I want to implement a pattern where I'll send only 1 message at once after the split, then sending the next after 1 message is acknowledged by the receiving system.
So instead of sending the 10 messages at once, I want
Send 1 message
Wait for the acknowledgment of that message
Send the next
Wait again
And so on..
How to implement such behavior?
I looked at the documentation, but none of the EIP components seem to fulfill that need?
Thanks for any input
You can have an intermediate seda queue with only one thread.
from("mock:some-route")
.split().method("splitterBean", "split")
.to("seda:your-seda-queue?waitForTaskToComplete=Always&timeout=0");
from("seda:your-seda-queue?waitForTaskToComplete=Always&timeout=0")
.to(ExchangePattern.InOut, "jms:some-thing");
By default, a seda queue will have a single consuming thread, and will block the calling thread until a consumer becomes available. More on seda details here
Saying that, your sending to a jms topic, which is really what you should be using to queue up your requests instead of a seda queue. You should look into implementing this logic asynchronously, and waiting on a reply topic rather than using a timeout.
I have a system that reads messages from a IBM MQ. From this queue we sometimes get corrupt data. If a message has failed the next time it is read the listener will wait 5 minutes before proceeding. When a message has failed 3 times it is put on a Backoff-queue. The JMSContainerFacotry is set up to run max 10 in parallell. This means if we get a big batch of corrupt messages these messages take 10+ minutes each to reach the Backoff-queue (which is ok), but they end up consuming all the threads so other messages must wait for all the corrupt to reach the Backoff-queue before being processed. I want to remove the wait() from the listener so that this doesnt block threads, but that failed messages still wait 5 minutes before popping back up.
I know that with ActiveMQ you can tell the queue to wait 5 minutes before popping the same message again, with IBM MQ I dont have that possibility. Is it possible to put failed messages to the back off the queue? Now failed messages seems to pop in front of new ones. Or set the jmspriority on failed messages lower. So that if there is messages that have yet to fail will get popped from the queue before those that have failed?
My listener is about so:
#JMSListener()
public void listen(Object message) {
TextMessage textMessage = (TextMessage) message;
if (textMessage.getIntProperty("JMSXDeliveryCount") > 1) {
//pause for 5 minutes
}
//buisness logic
}
The in-queue is a IBM MQ. AS is Jboss 6.
Not sure why you're pausing for 5 minutes if you get a poison message. If you know it's bad why aren't you either throwing it away or passing it to a backout queue for further investigation? See JMS poison message handling:
http://www.ibm.com/support/knowledgecenter/SSFKSJ_9.0.0/com.ibm.mq.dev.doc/q032280_.htm
More concerning, why are you getting so many 'bad' messages?
I know that with ActiveMQ you can tell the queue to wait 5 minutes
before popping the same message again, with IBM MQ I dont have that
possibility.
The option of retry interval between attempts need to be set on the MDB listener side, since it is acting as client. But not on WMQ side.
Is it possible to put failed messages to the back off the queue?
You wanted to place the message back on the queue or place the poisonous messages at the end of the same queue ? we can't move poison messages to the end of the same queue
There are only three options to handle poisonous messages:
Place the message back on the queue
re-route the message either to BOQ or DLQ
Purge or discard it permanently.
I am using ActiveMQ version 5.7.x,
I am having one ActiveMQ queue to which the listener listens.
Queue has a ConnectionFactory whose redeliveryPolicy is set to 3, intialRedeliveryDelay set to 5000.
Queue have some good messages and bad messages. While listening to such queue, when bad messages come, they are tried 3 times with the wait time of 5000 millis, but then the good messages are blocked for that much time span.
What I want is, during the wait time of 5000 millis for each retry, the processing of good messages should continue and should not wait for bad message processing.
For this I tried 1 attribute of connectionFactory, i.e. nonBlockingRedelivery set to true.
But nonBlockingRedelivery is not working.
Is there any other way to do this?
You can always have a retransmission queue for failing messages.
That is, receive messages from a main queue (no redelivery) and if you get an exception, put the message on a redelivery queue.
Let your application listen on both queues and do the same logic to both messages. It should simply be two message listeners invoking the same method. One with redelivery and one without, but with slightly different error handling.
Please look at the description of that attribute in ActiveMQConnectionFactory,
It says what I exactly what I wanted..
"When true a MessageConsumer will not stop Message delivery before re-delivering Messages from a rolled back transaction. This implies that message order will not be preserved and also will result in the TransactedIndividualAck option to be enabled"
But same is not working..!!!
Can you please look into it please?