Develop Reference

Kafka partition blocked when infra problem in a single instance of application

I have a problem with some micro service when running the microservice with kubernetes with many PODs.
I use the Manual commit strategy so I should acknowledge or not every message.
All the instance of the application belong the same kafka group. And the topic have at least 20 partition divided between the PODs. When consuming the message the listener make a call to a extern component (like a rest API by WebClient or RestTemplate, or a kafka producer of a different topic) . The Kafka consumer look like this:
#KafkaListener(topics = "topic")
#Trace
public void listen(#Payload Object message, , Acknowledgment acknowledgment)) {
try {
api.call(message);
acknowledgment.acknowledge();
} catch (InfraException e) {
acknowledgment.nack(1000);
}
But sometime this external component have infra problems and is not available. The problem usually happens when for some reason a single POD have connectivity issues. As the message is not acknowledge, he continue to be consumed, what is good. But the problem is that the message continue to be send to the same problematic instance of the application and is never redirected to another 'healthy' consumer.. As the consumer is able to get message from kafka and send heartbeats it is never considered a problematic consumer for kafka even after the rebalancing.
There some strategy or something we can do in the config to solve this problem or avoid the partition to be blocked?
Thank you for your attention so far.

Spring Cloud Stream with RabbitMQ binder, how to apply #Transactional?

I have a Spring Cloud Stream application that receives events from RabbitMQ using the Rabbit Binder. My application can be summarized as this:
#Transactional
#StreamListener(MySink.SINK_NAME)
public void processEvents(Flux<Event> events) {
// Transform events and store them in MongoDB using
// spring-boot-data-mongodb-reactive
...
}
The problem is that it doesn't seem that #Transactional works with Spring Cloud Stream (or at least that's my impression) since if there's an exception when writing to MongoDB the event seems to have already been ack:ed to RabbitMQ and the operation is not retried.
Given that I want to achieve basically the same functionality as when using the #Transactional around a function with spring-amqp:
Do I have to manually ACK the messages to RabbitMQ when using Spring
Cloud Stream with the Rabbit Binder?
If so, how can I achieve this?

There are several issues here.
Transactions are not required for acknowledging messages
Reactor-based #StreamListener methods are invoked exactly once, just to set up the Flux so #Transactional on that method is meaningless - messages then flow through the flux so anything pertaining to individual messages has to be done within the context of the flux.
Spring Transactions are bound to the thread - Reactor is non-blocking; the message will be acked at the first handoff.
Yes, you would need to use manual acks; presumably on the result of the mongodb store operation. You would probably need to use Flux<Message<Event>> so you would have access to the channel and delivery tag headers.

How to Prevent Spring AMQP from Blocking on Unacked Messages?

I have a #RabbitListener annotated method for which Spring AMQP blocks after returning from the method. The underlying SimpleRabbitListenerContainerFactory uses AcknowledgeMode.MANUAL. I don’t want to acknowledge the message in the listener method, yet.
Is there any way to not have Spring AMQP block in such a scenario?
In more detail
I use a listener like this:
#RabbitListener(queues = "#{ #myQueue }")
void recordRequestsFromMyMessages(
#Payload MyMessage myMessagePayload,
#Header(AmqpHeaders.DELIVERY_TAG) long deliveryTag,
Channel channel) {
// record relevant parts of the given message and combine them with
// parts from previous/future messages
// DON'T acknowledge the consumed message, yet; instead only keep a
// record of the channel and the delivery tag
}
Since I batch/combine multiple messages before I actually process them (asynchronously) later, I don’t want to acknowledge the consumed message right away. Instead, I only want to do this once the messages have been successfully processed later.
With my current approach, Spring AMQP blocks after returning from calling the recordRequestsFromMyMessages method above and no further messages are consumed from the same queue anymore.
This SO answer suggests that batch processing should work, however, I’m not sure how.

It's not the container that's "blocking".
You need to increase the prefetchCount on the container (default 1) - the broker only allows that number of unacked messages to be outstanding.

Can I receive a message, prepare its response and send a message in a context of a single JMS transaction?

Studying to a exam I just found a question I can't answer either after looking in the web. The question is:
"Can a server receive a request using a JMS message, prepare its response (e.g access a database) and send the reply using again, JMS, i the scope of a single JMS transaction?"
I know we can send a transacted message or receive a message in the context of a transaction. I know we can group several message sends and receives in a single transaction protecting the whole interaction. However, JMS is designed to be asynchronous. So in theory I would need to have a transaction to send the message to the queue and a transaction to receive the message from the queue. Am I right or is it possible to have a SINGLE transaction for a send and receive?

Yes, transacted receivers can be implemented in jMS. They are implemented by controlling the acknowledgemode of the communication: if all transactional operations are successful, the received message will be acknowledged to the broker, but in case of failure it does not happen, so the message can get redelivered.
This article explains this in more details:
Both message producers and message consumers may use transacted
sessions. [...]
With message consumers, transacted sessions control message
acknowledgment. The consumer can receive multiple messages just like
the CLIENT_ACKNOWLEDGE mode. When the associated transaction is
committed, the JMS implementation acknowledges all messages received
in the associated transaction. If the transaction aborts, the JMS
implementation returns the messages to the associated queue or topic.

Camel consume single message and stop, transacted

I am trying to use Camel to consume a single message from a JMS queue in a transacted manner. Specifically in a flow like this:
Wait until message is published on JMS queue
Try to consume and process the single message
If processing fails (exception occurs), rollback the consumption
If the processing passes, acknowledge and stop consuming anymore messages
Later in the application lifecycle, another process triggers consumption to start again from (1)
At first I tried to do this using a polling consumer, using the ConsumerTemplate, but I can't figure out if its possible to do this transactionally - it seems like the transaction is internal to the ConsumerTemplate so regardless of what I do the message is already acknowledged as consumed by the time the ConsumerTemplate returns.
Can I do this using the ConsumerTemplate? Can I do this using Camel and if so what is the best approach (Simple examples would be appreciated)?

I ended up using the pollEnrich dsl to achieve this. For example my route builder looks like:
from("direct:service-endpont").transacted("PROPOGATION_REQUIRED").setExchangePattern(ExchangePattern.InOut).pollEnrich("activemq:test-queue").bean(myHandler);
I use the direct endpoint as a service, sending a "request" message to the direct endpoint polls the jms queue for a single message (blocking if required). The transaction started extends to the pollEnrich so if, for example, myHandler bean fails then the message taken during the pollEnrich is not consumed and left on the queue.