I have a #RabbitListener annotated method for which Spring AMQP blocks after returning from the method. The underlying SimpleRabbitListenerContainerFactory uses AcknowledgeMode.MANUAL. I don’t want to acknowledge the message in the listener method, yet.
Is there any way to not have Spring AMQP block in such a scenario?
In more detail
I use a listener like this:
#RabbitListener(queues = "#{ #myQueue }")
void recordRequestsFromMyMessages(
#Payload MyMessage myMessagePayload,
#Header(AmqpHeaders.DELIVERY_TAG) long deliveryTag,
Channel channel) {
// record relevant parts of the given message and combine them with
// parts from previous/future messages
// DON'T acknowledge the consumed message, yet; instead only keep a
// record of the channel and the delivery tag
}
Since I batch/combine multiple messages before I actually process them (asynchronously) later, I don’t want to acknowledge the consumed message right away. Instead, I only want to do this once the messages have been successfully processed later.
With my current approach, Spring AMQP blocks after returning from calling the recordRequestsFromMyMessages method above and no further messages are consumed from the same queue anymore.
This SO answer suggests that batch processing should work, however, I’m not sure how.
It's not the container that's "blocking".
You need to increase the prefetchCount on the container (default 1) - the broker only allows that number of unacked messages to be outstanding.
Related
We have a requirement where we are consuming messages from one topic then there is some enrichment happening and then we are publishing the message to another topic. below are the events
Consumer - Consume the message
Enrichment - Enriched the consumed message
Producer - Published Enriched message to other topic
I am using Spring cloud kafka binder and things are working fine. suddenly we observed that producer is sending duplicate message to the topic and then we made Producer is idempotent. We have autocommitOffSet to false for better control. Below is what we are doing in the method
#StreamListener("INPUT")
#SendTo("OUTPUT")
public void consumer(Message message){
String inputMessage = message.getPayload.toString();
String enrichMessage = // Enrichment on inputMessage
return enrichMessage;
}
We observed if ack.acknowledge() failed due to some issue, Message still sent to the outbound channel. How can we handle entire consumer/producer as part of one transaction so that if acknowledge fail message will not sent to the topic.
I have set below transaction properties as well
spring.cloud.stream.kafka.binder.transaction.transactionIdPrefix=TX-
spring.cloud.stream.kafka.binder.transaction.producer.configuration.ack=all
spring.cloud.stream.kafka.binder.transaction.producer.configuration.retries=1
spring.cloud.stream.kafka.bindings.input.consumer.autoCommitOffset=true
spring.cloud.stream.kafka.bindings.input.consumer.enableDlq=true
spring.cloud.stream.kafka.bindings.input.consumer.dlqName=error.topic
spring.cloud.stream.kafka.bindings.input.consumer.autoCommitOnError=true
If there is any example available that would be really helpful.
Cheers
You need to make the binder transactional. See the documentation
https://docs.spring.io/spring-cloud-stream-binder-kafka/docs/3.1.4/reference/html/spring-cloud-stream-binder-kafka.html#_kafka_binder_properties
spring.cloud.stream.kafka.binder.transaction.transactionIdPrefix
Enables transactions in the binder. See transaction.id in the Kafka documentation and Transactions in the spring-kafka documentation. When transactions are enabled, individual producer properties are ignored and all producers use the spring.cloud.stream.kafka.binder.transaction.producer.* properties.
Default null (no transactions)
Note that consumers on the output topic must be configured with isolation.level=read_committed to avoid receiving rolled-back records.
I am trying to use Camel to consume a single message from a JMS queue in a transacted manner. Specifically in a flow like this:
Wait until message is published on JMS queue
Try to consume and process the single message
If processing fails (exception occurs), rollback the consumption
If the processing passes, acknowledge and stop consuming anymore messages
Later in the application lifecycle, another process triggers consumption to start again from (1)
At first I tried to do this using a polling consumer, using the ConsumerTemplate, but I can't figure out if its possible to do this transactionally - it seems like the transaction is internal to the ConsumerTemplate so regardless of what I do the message is already acknowledged as consumed by the time the ConsumerTemplate returns.
Can I do this using the ConsumerTemplate? Can I do this using Camel and if so what is the best approach (Simple examples would be appreciated)?
I ended up using the pollEnrich dsl to achieve this. For example my route builder looks like:
from("direct:service-endpont").transacted("PROPOGATION_REQUIRED").setExchangePattern(ExchangePattern.InOut).pollEnrich("activemq:test-queue").bean(myHandler);
I use the direct endpoint as a service, sending a "request" message to the direct endpoint polls the jms queue for a single message (blocking if required). The transaction started extends to the pollEnrich so if, for example, myHandler bean fails then the message taken during the pollEnrich is not consumed and left on the queue.
I have a rabbitmq listener as a separate class and JSF 2 managed bean.
In my bean I send a message and need to wait for result. I can't use sendAndReceive... because I send the message to one queue but receive from another queue, so I assign correlationId before sending.
So I need to wait asynchronously, I need to wait until right message comes to the listener. How to do it in rmq?
Looking at javadoc and source of RabbitTemplate it seems that he waits for response in reply queue. Do you set 'reply-to' property in your messages? If yes, then RabbitTemplate sendAndReceive methods should wait for response in 'reply-to' queue. Be sure to populate replyTo field correctly and test it.
Side note:
In RabbitMQ you do not send messages to the queue.
You send messages to the exchanges. Exchanges are routing messages to the queue(s) using bindings. With default or direct exchange type it looks like you send directly to the queue, but this is over-simplification.
See https://www.rabbitmq.com/tutorials/amqp-concepts.html for details.
Edit:
It seems there are some fix for that in AMQP 1.4.5.RELEASE
https://spring.io/blog/2015/05/08/spring-amqp-1-4-5-release-and-1-5-0-m1-available
Configurable Exchange/Routing Key for Replies
Previously, when using request/reply messaging with the
RabbitTemplate, replies were routed to the default exchange and routed
with the queue name. It is now possible to supply a reply-address with
the form exchange/routingKey to route using a specific exchange and
routing key.
I am new to JMS. As far as I understood Consumers are capable of picking messages from queue/topic. So why do you need a MessageListener because Consumers will know when they have picked up messages? What is the practical use of such a MessageListener?
Edit:From the Javadoc of MessageListener:
A MessageListener object is used to receive asynchronously delivered
messages.
Each session must insure that it passes messages serially to the
listener. This means that a listener assigned to one or more consumers
of the same session can assume that the onMessage method is not called
with the next message until the session has completed the last call.
So I am confused between the usage of the terms asynchronously and serially together. How do these two terms relate in describing the feature of MessageListener?
The difference is that MessageConsumer is used to receive messages synchronously:
MessageConsumer mc = s.createConsumer(queue);
Message msg = mc.receive();
For asynchronous delivery, we can register a MessageListener object with a message consumer:
mc.setMessageListener(new MessageListener() {
public void onMessage(Message msg) {
...
}
});
from the docs:
For synchronous receipt, a client can request the next message from a message consumer using one of its receive methods.
For asynchronous delivery, a client can register a MessageListener object with a message consumer.
One major difference as per my knowledge not stated in others answers is that MessageConsumer can make use of MessageSelectors and hence has capability to consume messages that it's interested in, where as MessageListener will listen to all messages.
From the J2EE tutorial doc http://docs.oracle.com/javaee/5/tutorial/doc/bnceh.html
JMS Message Selectors
If your messaging application needs to filter the messages it receives, you can use a JMS API message selector, which allows a message consumer to specify the messages it is interested in. Message selectors assign the work of filtering messages to the JMS provider rather than to the application.
We've a Spring JMS message listener container for receiving messages asynchronously. Using DefaultMessageListenerContainer and in sessionTransacted mode. I understand being in sessionTransacted mode means in case of an exception the message will be put back into the queue. But how can I make sure the message won't be deleted from the queue even if the receiver (which is picked the message) crashes or just the machine running it looses power?
At first I thought CLIENT_ACKNOWLEDGE acknowledge mode should save me, but apparently it's not the case, Spring calls .acknowledge() no matter what.
So here's my question, how can I guarantee the delivery? Using a custom MessageListenerContainer? Using a transaction manager?
Use a transacted session and indicate successful message processing by invoking the Session class's commit() method.
Check the section 19.4.5. Processing messages within transactions for the configuration. (you can use a DefaultMessageListenerContainer). Depending on what you're doing with the messages, you may need a JTA transaction manager.
Spring message listener with Client_Acknowledge mode will acknowledge the message when the client calls message.acknowledge().
However, if after successful execution of the message, the consumer does not find any acknowledgement from the client side, spring assumes the execution was successful and acknowledges the message.
If at any point of time, consumer got an exception while processing the message, spring listener needs to know some exception has occured in order to redeliver the message to the queue for another consumer thread to pick it up. If you're catching the exception, spring assumes everything was handled and execution was smooth and hence acknowledges the message.
Spring message listener only allows to throw JMS exception from onMessage listener. Catching your custom exception and throwing a JMS exception from listener (after logging the error for future reference) will allow you to redeliver the message.
or you can use Session.AUTO_ACKNOWLEDGE with nontransacted session, see quote below from this article
A message is automatically
acknowledged when it successfully
returns from the receive() method. If
the receiver uses the MessageListener
interface, the message is
automatically acknowledged when it
successfully returns from the
onMessage() method. If a failure
occurs while executing the receive()
method or the onMessage() method, the
message is automatically redelivered.
The JMS provider carefully manages
message redelivery and guarantees
once-only delivery semantics.