Develop Reference

Republish message to same queue with updated headers after automatic nack in Spring AMQP

I am trying to configure my Spring AMQP ListenerContainer to allow for a certain type of retry flow that's backwards compatible with a custom rabbit client previously used in the project I'm working on.
The protocol works as follows:
A message is received on a channel.
If processing fails the message is nacked with the republish flag set to false
A copy of the message with additional/updated headers (a retry counter) is published to the same queue
The headers are used for filtering incoming messages, but that's not important here.
I would like the behaviour to happen on an opt-in basis, so that more standardised Spring retry flows can be used in cases where compatibility with the old client isn't a concern, and the listeners should be able to work without requiring manual acking.
I have implemented a working solution, which I'll get back to below. Where I'm struggling is to publish the new message after signalling to the container that it should nack the current message, because I can't really find any good hooks after the nack or before the next message.
Reading the documentation it feels like I'm looking for something analogous to the behaviour of RepublishMessageRecoverer used as the final step of a retry interceptor. The main difference in my case is that I need to republish immediately on failure, not as a final recovery step. I tried to look at the implementation of RepublishMessageRecoverer, but the many of layers of indirection made it hard for me to understand where the republishing is triggered, and if a nack goes before that.
My working implementation looks as follows. Note that I'm using an AfterThrowsAdvice, but I think an error handler could also be used with nearly identical logic.
/*
MyConfig.class, configuring the container factory
*/
#Configuration
public class MyConfig {
#Bean
// NB: bean name is important, overwrites autoconfigured bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(
ConnectionFactory connectionFactory,
Jackson2JsonMessageConverter messageConverter,
RabbitTemplate rabbitTemplate
) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setMessageConverter(messageConverter);
// AOP
var a1 = new CustomHeaderInspectionAdvice();
var a2 = new MyThrowsAdvice(rabbitTemplate);
Advice[] adviceChain = {a1, a2};
factory.setAdviceChain(adviceChain);
return factory;
}
}
/*
MyThrowsAdvice.class, hooking into the exception flow from the listener
*/
public class MyThrowsAdvice implements ThrowsAdvice {
private static final Logger logger = LoggerFactory.getLogger(MyThrowsAdvice2.class);
private final AmqpTemplate amqpTemplate;
public MyThrowsAdvice2(AmqpTemplate amqpTemplate) {
this.amqpTemplate = amqpTemplate;
}
public void afterThrowing(Method method, Object[] args, Object target, ListenerExecutionFailedException ex) {
var message = message(args);
var cause = ex.getCause();
// opt-in to old protocol by throwing an instance of BusinessException in business logic
if (cause instanceof BusinessException) {
/*
NB: Since we want to trigger execution after the current method fails
with an exception we need to schedule it in another thread and delay
execution until the nack has happened.
*/
new Thread(() -> {
try {
Thread.sleep(1000L);
var messageProperties = message.getMessageProperties();
var count = getCount(messageProperties);
messageProperties.setHeader("xb-count", count + 1);
var routingKey = messageProperties.getReceivedRoutingKey();
var exchange = messageProperties.getReceivedExchange();
amqpTemplate.send(exchange, routingKey, message);
logger.info("Sent!");
} catch (InterruptedException e) {
logger.error("Sleep interrupted", e);
}
}).start();
// NB: Produce the desired nack.
throw new AmqpRejectAndDontRequeueException("Business logic exception, message will be re-queued with updated headers", cause);
}
}
private static long getCount(MessageProperties messageProperties) {
try {
Long c = messageProperties.getHeader("xb-count");
return c == null ? 0 : c;
} catch (Exception e) {
return 0;
}
}
private static Message message(Object[] args) {
try {
return (Message) args[1];
} catch (Exception e) {
logger.info("Bad cast parse", e);
throw new AmqpRejectAndDontRequeueException(e);
}
}
}
Now, as you can imagine, I'm not particularly pleased with the indeterminism of scheduling a new thread with a delay.
So my question is simply, is there any way I could produce a deterministic solution to my problem using the provided hooks of the ListenerContainer ?

Your current solution risks message loss; since you are publishing on a different thread after a delay. If the server crashes during that delay, the message is lost.
It would be better to publish immediately to another queue with a TTL and dead-letter configuration to republish the expired message back to the original queue.
Using the RepublishMessageRecoverer with retries set to maxattempts=1 should do what you need.

Java RabbitMQ consumer.nextMessage always gets same message

We are using Java rabbitMq with spring boot in a distributed service architecture. One service gets an HTTP request and forwards it to an unkown queue for processing. At the same time it has to wait for a response on another queue before it can terminate the HTTP request. (It's a preview request that gets its work done by a renderer).
There can be more than one instance of ServiceA (the HTTP Interface) and ServiceB (the renderer) so with every preview message we also send a unique ID to be used as routing key.
I'm having trouble with the BlockingConsumer. Whenever I call consumer.nextMessage() I get the same message over and over again. This is doubly weird, as for one it should be ACKed and removed from the queue and for another the consumer shouldn't even bother with it as the unique ID we used is no longer bound to the queue. nextMessage even returns before the renderer service is done and has sent its done message back.
Here's the simplified setup:
general
All services use a global DirectExchange for all messages
#Bean
public DirectExchange globalDirectExchange() {
return new DirectExchange(EXCHANGE_NAME, false, true);
}
ServiceA (handles the HTTP request):
private Content requestPreviewByKey(RenderMessage renderMessage, String previewKey) {
String renderDoneRoutingKey= UUID.randomUUID().toString();
renderMessage.setPreviewDoneKey(renderDoneId);
Binding binding = BindingBuilder.bind(previewDoneQueue).to(globalDirectExchange)
.with(renderDoneRoutingKey);
try {
amqpAdmin.declareBinding(binding);
rabbitProducer.sendPreviewRequestToKey(renderMessage, previewKey);
return getContentBlocking();
} catch (Exception e) {
logErrorIfDebug(type, e);
throw new ApiException(BaseErrorCode.COMMUNICATION_ERROR, "Could not render preview");
} finally {
amqpAdmin.removeBinding(binding);
}
}
private Content getContentBlocking() {
BlockingQueueConsumer blockingQueueConsumer = new BlockingQueueConsumer(rabbitMqConfig.connectionFactory(), new DefaultMessagePropertiesConverter(), new ActiveObjectCounter<>(), AcknowledgeMode.AUTO, true, 1, PREVIEW_DONE_QUEUE);
try {
blockingQueueConsumer.start();
Message message = blockingQueueConsumer.nextMessage(waitForPreviewMs);
if (!StringUtils.isEmpty(message)) {
String result = new String(message.getBody());
return JsonUtils.stringToObject(result, Content.class);
}
throw new ApiException("Could not render preview");
} catch (Exception e) {
logError(e);
throw new ApiException("Could not render preview");
} finally {
blockingQueueConsumer.stop();
}
}
Service B
I'll spare you most of the code. My log says everything is going well and as soon as its done the service sends the correct message to the UUID key that was sent with the initial render request.
public void sendPreviewDoneMessage(Content content, String previewDoneKey) {
String message = JsonUtils.objectToString(content);
rabbitTemplate.convertAndSend(globalDirectExchange, previewDoneKey, message);
}
The whole thing works... Once...
The real issues seems to be the consumer setup. Why do I keep getting the same (first) message from the queue when I use nextMessage().
Doesn't creating and removing a Bindung ensure, that only messages bound to that routingKey are even received in that instance? And doesn't nextMessage() acknowledge the message and remove it from the queue?!
Thank's a lot for bearing with me and even more for any helpful answer!

BlockingQueueConsumer is not designed to be used directly; it is a component of the SimpleMessageListenerContainer, which will take care of acking the message after it has been consumed by a listener (the container calls commitIfNecessary).
There may be other unexpected side effects of using this consumer directly.
I strongly advise using the listener container to consume messages.
If you just want to receive messages on demand, use a RabbitTemplate receive() or receiveAndConvert() method instead.

Spring Boot rabbit mq spring.rabbitmq.listener.simple.concurrency never works

In SpringBoot rabbit mq project,I have the configuration like that.
#EnableRabbit
#Configuration
public class MQConfig {
private final String primaryQueueName;
private final String deadLetterQueueName;
#Autowired
private ConnectionFactory cachingConnectionFactory;
public MQConfig(
#Value("${primaryQueue.name}")String primaryQueueName,
#Value("${deadLetterQueue.name}")String deadLetterQueueName){
this.primaryQueueName=primaryQueueName;
this.deadLetterQueueName=deadLetterQueueName;
}
#Bean
public Queue primaryQueue() {
Map<String, Object> args = new HashMap<String, Object>();
// The default exchange
args.put("x-dead-letter-exchange", "");
// Route to the incoming queue when the TTL occurs
args.put("x-dead-letter-routing-key", deadLetterQueueName);
// TTL 500 seconds
args.put("x-message-ttl", 300000);
return new Queue(primaryQueueName, false, false, false, args);
}
#Bean
public Queue deadLetterQueue() {
return new Queue(deadLetterQueueName);
}}
And my listener looks like,
#Component
public class Receiver {
private static final Logger LOGGER = LoggerFactory.getLogger(Receiver.class);
#Autowired
private JavaMailSender mailSender;
#RabbitListener(queues = "${primaryQueue.name}")
public void receiveMessage(SimpleMailMessage message){
LOGGER.info("Sending email message : "+message);
try {
mailSender.send(message);
}catch(Exception e) {
LOGGER.error("Failed to deliver email",e);
throw e;
}
}}
So far so good, the messages come to primary queue, the consumer consumes them and they are gone to dead letter queue if for some reason the consumer is not able to consume them in given TTL 300000.
Now, I want to configure the listener even further with spring application.properties, but they have no impact on it at all.
E.g. I set spring.rabbitmq.listener.simple.concurrency=3, still it creates a single consumer, explicitly setting it in code works however. Nether works other settings,
spring.rabbitmq.listener.simple.acknowledge-mode= # Acknowledge mode of container.
spring.rabbitmq.listener.simple.auto-startup=true # Start the container automatically on startup.
spring.rabbitmq.listener.simple.concurrency= # Minimum number of consumers.
spring.rabbitmq.listener.simple.default-requeue-rejected= # Whether or not to requeue delivery failures; default `true`.
spring.rabbitmq.listener.simple.idle-event-interval= # How often idle container events should be published in milliseconds.
spring.rabbitmq.listener.simple.max-concurrency= # Maximum number of consumers.
spring.rabbitmq.listener.simple.prefetch= # Number of messages to be handled in a single request. It should be greater than or equal to the transaction size (if used).
spring.rabbitmq.listener.simple.retry.enabled=false # Whether or not publishing retries are enabled.
spring.rabbitmq.listener.simple.retry.initial-interval=1000 # Interval between the first and second attempt to deliver a message.
spring.rabbitmq.listener.simple.retry.max-attempts=3 # Maximum number of attempts to deliver a message.
spring.rabbitmq.listener.simple.retry.max-interval=10000 # Maximum interval between attempts.
spring.rabbitmq.listener.simple.retry.multiplier=1.0 # A multiplier to apply to the previous delivery retry interval.
spring.rabbitmq.listener.simple.retry.stateless=true # Whether or not retry is stateless or stateful.
spring.rabbitmq.listener.simple.transaction-size=
https://docs.spring.io/spring-boot/docs/current/reference/html/common-application-properties.html
Any ideas? Am I configuring something incorrectly ?

You reference the docs for the current version of Spring Boot. Is this what you are using, or something older? This property used to be spring.rabbitmq.listener.concurrency without the "simple" in there.
Double check the version, for example: https://docs.spring.io/spring-boot/docs/1.4.3.RELEASE/reference/html/common-application-properties.html

How to create Processor with Transaction and DLQ with Rabbit binding?

I'm just starting to learn Spring Cloud Streams and Dataflow and I want to know one of important use cases for me. I created example processor Multiplier which takes message and resends it 5 times to output.
#EnableBinding(Processor.class)
public class MultiplierProcessor {
#Autowired
private Source source;
private int repeats = 5;
#Transactional
#StreamListener(Processor.INPUT)
public void handle(String payload) {
for (int i = 0; i < repeats; i++) {
if(i == 4) {
throw new RuntimeException("EXCEPTION");
}
source.output().send(new GenericMessage<>(payload));
}
}
}
What you can see is that before 5th sending this processor crashes. Why? Because it can (programs throw exceptions). In this case I wanted to practice fault prevention on Spring Cloud Stream.
What I would like to achieve is to have input message backed in DLQ and 4 messages that were send before to be reverted and not consumed by next operand (just like in normal JMS transaction). I tried already to define following properties in my processor project but without success.
spring.cloud.stream.bindings.output.producer.autoBindDlq=true
spring.cloud.stream.bindings.output.producer.republishToDlq=true
spring.cloud.stream.bindings.output.producer.transacted=true
spring.cloud.stream.bindings.input.consumer.autoBindDlq=true
Could you tell me if it possible and also what am I doing wrong? I would be overwhelmingly thankful for some examples.

You have several issues with your configuration:
missing .rabbit in the rabbit-specific properties)
you need a group name and durable subscription to use autoBindDlq
autoBindDlq doesn't apply on the output side
The consumer has to be transacted so that the producer sends are performed in the same transaction.
I just tested this with 1.0.2.RELEASE:
spring.cloud.stream.bindings.output.destination=so8400out
spring.cloud.stream.rabbit.bindings.output.producer.transacted=true
spring.cloud.stream.bindings.input.destination=so8400in
spring.cloud.stream.bindings.input.group=so8400
spring.cloud.stream.rabbit.bindings.input.consumer.durableSubscription=true
spring.cloud.stream.rabbit.bindings.input.consumer.autoBindDlq=true
spring.cloud.stream.rabbit.bindings.input.consumer.transacted=true
and it worked as expected.
EDIT
Actually, no, the published messages were not rolled back. Investigating...
EDIT2
OK; it does work, but you can't use republishToDlq - because when that is enabled, the binder publishes the failed message to the DLQ and the transaction is committed.
When that is false, the exception is thrown to the container, the transaction is rolled back, and RabbitMQ moves the failed message to the DLQ.
Note, however, that retry is enabled by default (3 attempts) so, if your processor succeeds during retry, you will get duplicates in your output.
For this to work as you want, you need to disable retry by setting the max attempts to 1 (and don't use republishToDlq).
EDIT3
OK, if you want more control over the publishing of the errors, this will work, when the fix for this JIRA is applied to Spring AMQP...
#SpringBootApplication
#EnableBinding({ Processor.class, So39018400Application.Errors.class })
public class So39018400Application {
public static void main(String[] args) {
SpringApplication.run(So39018400Application.class, args);
}
#Bean
public Foo foo() {
return new Foo();
}
public interface Errors {
#Output("errors")
MessageChannel errorChannel();
}
private static class Foo {
#Autowired
Source source;
#Autowired
Errors errors;
#StreamListener(Processor.INPUT)
public void handle (Message<byte[]> in) {
try {
source.output().send(new GenericMessage<>("foo"));
source.output().send(new GenericMessage<>("foo"));
throw new RuntimeException("foo");
}
catch (RuntimeException e) {
errors.errorChannel().send(MessageBuilder.fromMessage(in)
.setHeader("foo", "bar") // add whatever you want, stack trace etc.
.build());
throw e;
}
}
}
}
with properties:
spring.cloud.stream.bindings.output.destination=so8400out
spring.cloud.stream.bindings.errors.destination=so8400errors
spring.cloud.stream.rabbit.bindings.errors.producer.transacted=false
spring.cloud.stream.rabbit.bindings.output.producer.transacted=true
spring.cloud.stream.bindings.input.destination=so8400in
spring.cloud.stream.bindings.input.group=so8400
spring.cloud.stream.rabbit.bindings.input.consumer.transacted=true
spring.cloud.stream.rabbit.bindings.input.consumer.requeue-rejected=false
spring.cloud.stream.bindings.input.consumer.max-attempts=1

What can be the best approach to handle java.net.UnknownHostException for AWS users?

My application sends message to Amazon Simple Notification Service (SNS) topic but sometime (6/10) I get java.net.UnknownHostException:sqs.ap-southeast-1.amazonaws.com. The reason of exception is described in the amazon web services discussion forums, please look: https://forums.aws.amazon.com/thread.jspa?messageID=499290&#499290.
My problem is similar to what described in forums of amazon but my rate of publishing messages to topic is very dynamic. It can be 1 message/second or 1 message/minute or no message in an hour. I am looking for a cleaner, better and safe approach, which guaranties sending of message to SNS topic.
Description of problem in detail:
Topic_Arn= arn of SNS topic where application wants to publish message
msg = Message to send in topic
// Just a sample example which publish message to Amazon SNS topic
class SimpleNotificationService {
AmazonSNSClient mSnsClient = null;
static {
createSnsClient()
}
private void static createSnsClient() {
Region region = Region.getRegion(Regions.AP_SOUTHEAST_1);
AWSCredentials credentials = new
BasicAWSCredentials(AwsPropertyLoader.getInstance().getAccessKey(),
AwsPropertyLoader.getInstance().getSecretKey());
mSqsClient = new AmazonSQSClient(credentials);
mSqsClient.setRegion(region);
}
public void static publishMessage(String Topic_Arn, String msg) {
PublishRequest req = new PublishRequest(Topic_Arn, msg);
mSnsClient.publish(req);
}
}
class which calls SimpleNotificationService
class MessagingManager {
public void sendMessage(String message) {
String topic_arn = "arn:of:amazon:sns:topic";
SimpleNotificationService.publishMessage(topic_arn, message);
}
}
Please note that this is a sample code, not my actual code. Here can be class design issue but please ignore those if they are not related to problem.
My thought process says to have try-catch block inside sendMessage, so when we catch UnknownHostException then again retry but I am not sure how to write this in safer, cleaner and better way.
So MessagingManager class will look something like this:
class MessagingManager {
public void sendMessage(String message) {
String topic_arn = "arn:of:amazon:sns:topic";
try {
SimpleNotificationService.publishMessage(topic_arn, message);
} catch (UnknownHostException uhe) {
// I need to catch AmazonClientException as aws throws
//AmazonClientException when sees UnknownHostException.
// I am mentioning UnknownHostException for non-aws user to understand
// my problem in better way.
sendMessage(message); // Isn't unsafe? - may falls into infinite loop
}
}
}
I am open for answers like this: java.net.UnknownHostException: Invalid hostname for server: local but my concern is to dependent on solution at application code-level and less dependent on changes to machine. As my server application is going to run in many boxes (developer boxes, testing boxes or production boxes). If changes in machine host-files or etc is only guaranted solution then I prefer that to include with code level changes.

Each AWS SDK implements automatic retry logic. The AWS SDK for Java automatically retries requests, and you can configure the retry settings using the ClientConfiguration class.
Below is the sample example to create SNS client. It retries for 25 times if encounters UnKnownHostException. It uses default BackOff and retry strategy. If you want to have your own then you need to implement these two interfaces: http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/retry/RetryPolicy.html
private void static createSnsClient() {
Region region = Region.getRegion(Regions.AP_SOUTHEAST_1);
AWSCredentials credentials = new
BasicAWSCredentials(AwsPropertyLoader.getInstance().getAccessKey(),
AwsPropertyLoader.getInstance().getSecretKey());
ClientConfiguration clientConfiguration = new ClientConfiguration();
clientConfiguration.setMaxErrorRetry(25);
clientConfiguration.setRetryPolicy(new RetryPolicy(null, null, 25, true));
mSnsClient = new AmazonSNSClient(credentials, clientConfiguration);
mSnsClient.setRegion(region);
}

Have you considering looking into the JVM TTL for the DNS Cache?
http://docs.aws.amazon.com/AWSSdkDocsJava/latest//DeveloperGuide/java-dg-jvm-ttl.html