Develop Reference

Why KafkaTemplate does not close tansactional producers?

I have written a simple Kafka app with spring integration kafka 3.2.1.RELEASE and kafka-clients 2.5 to learn kafka transactions.
It recieves the messages from a topic and sends them to another topic. The beans.xml file is as follows
<int-kafka:message-driven-channel-adapter
listener-container="container"
auto-startup="true"
send-timeout="30000"
channel="channelA"/>
<bean id="container" class="org.springframework.kafka.listener.KafkaMessageListenerContainer" parent="kafkaMessageListenerContainerAbstract">
<constructor-arg>
<bean class="org.springframework.kafka.listener.ContainerProperties">
<constructor-arg
name="topics"
value="test"/>
<property name="transactionManager" ref="KafkaTransactionManager"/>
</bean>
</constructor-arg>
</bean>
.
.
.
<int-kafka:outbound-channel-adapter kafka-template="kafkaTemplate"
auto-startup="true"
channel="channelB"
topic="output"/>
<bean id="dbsenderTemplate" class="org.springframework.kafka.core.KafkaTemplate">
<constructor-arg>
<bean class="org.springframework.kafka.core.DefaultKafkaProducerFactory">
<constructor-arg>
<map>
<entry key="value.serializer" value="org.apache.kafka.common.serialization.StringSerializer"/>
<entry key="key.serializer" value="org.apache.kafka.common.serialization.StringSerializer"/>
<entry key="bootstrap.servers" value="localhost:9092"/>
</map>
</constructor-arg>
<property name="transactionIdPrefix" value="mytest-"/>
<property name="producerPerConsumerPartition" value="false"/>
</bean>
</constructor-arg>
</bean>
The code that starts the app is as follows:
GenericXmlApplicationContext tempContext = new GenericXmlApplicationContext("beans.xml");
tempContext.close();
//POINT A.
try {
Thread.sleep(60000);
} catch (InterruptedException e) {
e.printStackTrace();
}
GenericXmlApplicationContext context = new GenericXmlApplicationContext();
context.load("beans.xml");
context.refresh();
//POINT B
At POINT A I just closed the context to check which beans are closed, and put a 60 seconds sleep to have time to check the JMX console. I noticed that even though the context is closed but the producer is still registered in JMX. After that I traced the code and noticed that on context closing the KafkaTemplate calls the following code:
public void flush() {
Producer<K, V> producer = getTheProducer();
try {
producer.flush();
}
finally {
closeProducer(producer, inTransaction());
}
}
protected void closeProducer(Producer<K, V> producer, boolean inTx) {
if (!inTx) {
producer.close(this.closeTimeout);
}
}
It means it creates a producer but because it is transactional it will not be closed.
This behaviour makes that runnning the context again on POINT B and sending the message cause the javax.management.InstanceAlreadyExistsException: kafka.producer:type=app-info,id=producer-mytest-0 Exception.
Why the KafkaTemplate does not close these producers?
And another question is what happens to these producers when a new KafkaTemplate is created on POINT B?
The last question is if I change the producerPerConsumerPartition property to true the mentioned app still registers producer Mbean with producer-mytest-0 and does not follow the groupid.topic.partition pattern in naming. Is it a correct behaviour?
UPDATES:
I understood when the KafkaTemplate executeInTransaction is called. At the finally block it calls the close on the producer and as it is a logical close, the following code is called on the CloseSafeProducer and put it in the cache:
if (!this.cache.contains(this)
&& !this.cache.offer(this)) {
this.delegate.close(closeTimeout);
}
This makes when the context is closed the destroy method of DefaultKafkaProducerFactory clears the cache and closes the producer physically. But in my situation application context is created but before consume and producing any message the context is closed, only the flush method of KafkaTemplate is called internally that force it to create a transactional producer but does not put it in the cache. As I didn't start a producer and KafkaTemplate do it on flush, is not it good that DefaultKafkaProducerFactory put them in cache before using them?

The producer cannot be closed if this template operation is participating in a transaction that was started outside of the template.
Even when closed, it is only "logically" closed - cached for reuse by another operation.
Is it a correct behaviour?
Yes, for producer-initiated transactions; the alternative name is used when a consumer initiates the transaction.
The InstanceAlreadyExistsException problem is simply because you are creating two application contexts with identical configuration. Why are you doing that?

How to organize Spring Integration retry for a method with a list that decreases with every retry?

There is the following Spring Integration configuration with Retry:
<int:chain input-channel="sdCreationChannel" output-channel="debugLogger">
<int:poller fixed-delay="500" />
<int:filter ref="sdIntegrationExistingRequestSentFilter" method="filter"/>
<int:service-activator ref="sdCreationServiceImpl" method="processMessage">
<int:request-handler-advice-chain>
<ref bean="retryAdvice"/>
</int:request-handler-advice-chain>
</int:service-activator>
</int:chain>
<bean id="retryAdvice" class="org.springframework.integration.handler.advice.RequestHandlerRetryAdvice" >
<property name="retryTemplate">
<bean class="org.springframework.retry.support.RetryTemplate">
<property name="backOffPolicy">
<bean class="org.springframework.retry.backoff.ExponentialBackOffPolicy">
<property name="initialInterval" value="${integration.retry.initial.delay}"/>
<property name="multiplier" value="${integration.retry.backoff.multiplier}"/>
</bean>
</property>
<property name="retryPolicy">
<bean class="org.springframework.retry.policy.SimpleRetryPolicy">
<property name="maxAttempts" value="${integration.retry.max.attempts}" />
</bean>
</property>
</bean>
</property>
</bean>
The simplified Java code is as follows:
#Component("sdCreationServiceImpl")
public class SDCreationServiceImpl implements SDCreationService {
#Autowired
private NotifySD notifySD;
#Override
public void processMessage(IntegrationPayload integrationPayload) {
List<ConfirmationCode> sdConfCodes = findCodesFromPayLoad(integrationPayload);
notifySD.activateConfirmationCodes(sdConfCodes);
}
The problem with retrying this code is that the List sdConfCodes can be partially processed at every Retry, so every time we need to send for processing the less amount of elements. What is best way to organize this code?
Following the Artem Bilan suggestion (Thanks!) I created the 2nd method with variable list in SDCreationServiceImpl, i.e. activateConfirmationCodes and then in the XML specification pointed to this method as a method for sdCreationServiceImpl.
#Component("sdCreationServiceImpl")
public class SDCreationServiceImpl implements SDCreationService {
#Autowired
private NotifySD notifySD;
List<ConfirmationCode> sdConfCodes = new ArrayList<ConfirmationCode()>;
#Override
public void processMessage(IntegrationPayload integrationPayload) {
sdConfCodes = findCodesFromPayLoad(integrationPayload);
}
public void activateConfirmationCodes()
{
notifySD.activateConfirmationCodes(sdConfCodes);
}
And then the XML spec for service-activator is as follows:
<int:service-activator ref="sdCreationServiceImpl" method="activateConfirmationCodes">
<int:request-handler-advice-chain>
<ref bean="retryAdvice"/>
</int:request-handler-advice-chain>
</int:service-activator>
Yes, this method activateConfirmationCodes is invoked in Retry but the 1st method processMessage is not invoked at all. Is it possible to specify one method for invoking at the first try and other method for retrying?
Second with this design the list becomes the singleton and this can give problems in multithreading, correct?. Can this list be associated with a bean only for a particular message?

From big it isn't clear where is your problem. From other side let me share some my thoughts, perhaps I will guess your goal.
Having the List<ConfirmationCode> as a payload allows us to modify it at any time. So, let's assume we have list as 10 elements. On the first attempt we have processed 3 of them. The fourth has failed. We have to go to retry throwing some appropriate exception.
But we come back to the beginning of the retry-aware method, so with the same arguments. If we remove those successful items from the collection, the next retry iteration won't process them at all.
From one side you can achieve that distinguishing findCodesFromPayLoad() service and activateConfirmationCodes(), applying retry for the last one.
From other side you can mark items as processed in the activateConfirmationCodes(), so the next findCodesFromPayLoad(integrationPayload) won't return them.
In other words there are enough ways to modify collection without changing messages.

DefaultMessageListenerContainer, knowledge about the queue to listen on

Is there any possibility to know inside the onMessage method, which queue the MessageListener is listening to?
My Spring-config (a part of it):
<bean id="abstractMessageListenerContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer" abstract="true">
<property name="connectionFactory" ref="connectionFactory" />
<property name="maxConcurrentConsumers" value="5"/>
<property name="receiveTimeout" value="10000" />
</bean>
<bean class="org.springframework.jms.listener.DefaultMessageListenerContainer" parent="abstractMessageListenerContainer">
<property name="destinationName" value="MY.QUEUE" />
<property name="messageListener" ref="myMessageListener" />
</bean>
<bean id="myMessageListener" class="my.package.structure.ListenerClass"></bean>
My Listener Class:
public class ListenerClass implements MessageListener {
public void onMessage(Message msg) {
//where do I know from, on which queue the message has been written to?
}
}
Is there any out-of-the box solution? Or any custom solution to get the queue/destination-name?
Do need the queue in a subsequent batch-processing...

Easy. in trivial cases at least:
msg.getJMSDestination() will give you the destination as a javax.jms.Destination object. Typically .toString() returns something like: queue://MYQUEUENAME
However, in some JMS implementations, there might be multihop queues, such as a static pub/sub setup in WebSphere MQ where you might write your message to one queue and it will bounce around a route to end up in a completly different queue. Also, you might have the case of an ESB with logic in the middle that routes the message. In such cases, you will need to think twice before relying to much on the JMSDestination attribute. Otherwise, go ahead.

Multiple instances of a bean

I'm implementing a listener class which listens for some events and then processes them. If processing of this event goes well then this event is never notified again, but in case of any exception event will be again notified to MyBeanImplementation class after some time and it can again try processing it.
Following code works fine but since this event processing may take some time,
1. I want to have multiple listeners.
2. Limit number of calls to service, may be using threadpool.
How can I have multiple listeners which will process each event differently? I'm new to Spring and don't have much idea if this is even possible or not.
Heres is an example:
// Spring Configuration:
<bean id="MyBean" class="MyBeanImplementation">
// Sample Class
public class MyBeanImplementation implements EventListener {
#override
public processEvent(Event event) throws EventProcessFailureException {
try {
// Validate event
validateEvent(event);
// Call another service to store part of information from this event
// This service takes some time to return success
boolean success = makeCallToServiceAndStoreInfo(event);
if(!success) {
throw new EventProcessFailureException("Error storing event information!");
}
} catch (Exception e) {
throw new EventProcessFailureException(e);
}
}
}
Thanks

Basically, you can use Strategy pattern to introduce different listeners that re-act in a different way to a shared event.
<bean id="strategy1Listener" />
<bean id="strategy2Listener" />
<bean id="strategy3Listener" />
Then, you can introduce a composite listener to iterate through other listeners and pass through the event and allow them to process the event:
<bean id="compositeStrategyListener">
<property name="listeners">
<list>
<ref bean="strategy1Listener" />
<ref bean="strategy2Listener" />
<ref bean="strategy3Listener" />
</list>
</property>
</bean>
On the other side of the story, you have an object that generates/publishes the events:
<bean id="eventGenerator">
<property name="eventListener" ref="compositeStrategyListener" />
</bean>
So, now eventGenerator publishes the generated event to a compositeStrategyListner which iterates over the listeners it has and allows them to process the event in their own different way.
You can also take advantage of Spring Task Execution to configure how you need to run the task of event processing.

Why not using google guava eventbus library with spring together? It will handle every thing needed for listening events.

How to receive what was sent by convertAndSend?

I'm reading Spring Framework reference, chapter about JMS integration. There are some examples for sending text messages and asynchronously receiving them (by listeners). And there is also an example for JmsTemplate function convertAndSend which converts given object to a message. The reference says:
By using the converter, you and your application code can focus on the business object that is being sent or received via JMS and not be concerned with the details of how it is represented as a JMS message.
But there is no example for receiving such messages. They mention function receiveAndConvert but, unfortunately, it receives synchronously.
So how am I to receive it asynchronously? Must I be aware that when I convertAndSend a Map, the resulting message will be a MapMessage, and just check in my listener for this type of message and handle it? But they promised I'm not to be concerned with the details of how it is represented as a JMS message.
So is there a better way?

I know it's been a while since this was asked, but I had the same problem, solved it and wanted to give an explicit code example here.
Here's my MessageListener. This implements the onMessage(Message) method to intercept messages asynchronously.
package com.package.amqp;
import org.springframework.amqp.core.Message;
import org.springframework.amqp.core.MessageListener;
import org.springframework.amqp.support.converter.JsonMessageConverter;
import com.package.model.User;
public class TestListener implements MessageListener {
public void onMessage(Message message) {
JsonMessageConverter jmc = new JsonMessageConverter();
User u = (User)jmc.fromMessage(message);
System.out.println("received: " + u.getFirstName());
}
}
The messages are then converted using the standard JsonMessageConvertor in my case as this is the messageConvertor I plugged into my rabbitTemplate bean.
<bean id="rabbitConnectionFactory" class="org.springframework.amqp.rabbit.connection.SingleConnectionFactory">
<constructor-arg value="10.10.1.2"/>
<property name="username" value="guest"/>
<property name="password" value="guest"/>
</bean>
<bean class="org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer">
<property name="connectionFactory" ref="rabbitConnectionFactory"/>
<property name="queueName" value="queue.helloWorld"/>
<property name="messageListener" ref="someListener"/>
</bean>
<bean id="someListener" class="com.package.amqp.TestListener"></bean>
<bean id="rabbitTemplate" class="org.springframework.amqp.rabbit.core.RabbitTemplate">
<property name="connectionFactory" ref="rabbitConnectionFactory"/>
<property name="messageConverter">
<bean class="org.springframework.amqp.support.converter.JsonMessageConverter"/>
</property>
</bean>
Hope this helps someone!
Owen

While JmsTemplate provides basic synchronous receive methods, asynchronous reception is a whole lot more complicated, and is beyond the scope of JmsTemplate.
Asynchronous reception of JMS messages is done in Spring using Message Listener Containers, which asynchronously take messages from the JMS destination and pass them to your application. You can plug a MessageConverter in to your message listener container via a MessageListenerAdapter (plug the converter into the adapter, plug your application's listener into the adapter, then plug the adapter into the listener container).