I have written a simple Kafka app with spring integration kafka 3.2.1.RELEASE and kafka-clients 2.5 to learn kafka transactions.
It recieves the messages from a topic and sends them to another topic. The beans.xml file is as follows
<int-kafka:message-driven-channel-adapter
listener-container="container"
auto-startup="true"
send-timeout="30000"
channel="channelA"/>
<bean id="container" class="org.springframework.kafka.listener.KafkaMessageListenerContainer" parent="kafkaMessageListenerContainerAbstract">
<constructor-arg>
<bean class="org.springframework.kafka.listener.ContainerProperties">
<constructor-arg
name="topics"
value="test"/>
<property name="transactionManager" ref="KafkaTransactionManager"/>
</bean>
</constructor-arg>
</bean>
.
.
.
<int-kafka:outbound-channel-adapter kafka-template="kafkaTemplate"
auto-startup="true"
channel="channelB"
topic="output"/>
<bean id="dbsenderTemplate" class="org.springframework.kafka.core.KafkaTemplate">
<constructor-arg>
<bean class="org.springframework.kafka.core.DefaultKafkaProducerFactory">
<constructor-arg>
<map>
<entry key="value.serializer" value="org.apache.kafka.common.serialization.StringSerializer"/>
<entry key="key.serializer" value="org.apache.kafka.common.serialization.StringSerializer"/>
<entry key="bootstrap.servers" value="localhost:9092"/>
</map>
</constructor-arg>
<property name="transactionIdPrefix" value="mytest-"/>
<property name="producerPerConsumerPartition" value="false"/>
</bean>
</constructor-arg>
</bean>
The code that starts the app is as follows:
GenericXmlApplicationContext tempContext = new GenericXmlApplicationContext("beans.xml");
tempContext.close();
//POINT A.
try {
Thread.sleep(60000);
} catch (InterruptedException e) {
e.printStackTrace();
}
GenericXmlApplicationContext context = new GenericXmlApplicationContext();
context.load("beans.xml");
context.refresh();
//POINT B
At POINT A I just closed the context to check which beans are closed, and put a 60 seconds sleep to have time to check the JMX console. I noticed that even though the context is closed but the producer is still registered in JMX. After that I traced the code and noticed that on context closing the KafkaTemplate calls the following code:
public void flush() {
Producer<K, V> producer = getTheProducer();
try {
producer.flush();
}
finally {
closeProducer(producer, inTransaction());
}
}
protected void closeProducer(Producer<K, V> producer, boolean inTx) {
if (!inTx) {
producer.close(this.closeTimeout);
}
}
It means it creates a producer but because it is transactional it will not be closed.
This behaviour makes that runnning the context again on POINT B and sending the message cause the javax.management.InstanceAlreadyExistsException: kafka.producer:type=app-info,id=producer-mytest-0 Exception.
Why the KafkaTemplate does not close these producers?
And another question is what happens to these producers when a new KafkaTemplate is created on POINT B?
The last question is if I change the producerPerConsumerPartition property to true the mentioned app still registers producer Mbean with producer-mytest-0 and does not follow the groupid.topic.partition pattern in naming. Is it a correct behaviour?
UPDATES:
I understood when the KafkaTemplate executeInTransaction is called. At the finally block it calls the close on the producer and as it is a logical close, the following code is called on the CloseSafeProducer and put it in the cache:
if (!this.cache.contains(this)
&& !this.cache.offer(this)) {
this.delegate.close(closeTimeout);
}
This makes when the context is closed the destroy method of DefaultKafkaProducerFactory clears the cache and closes the producer physically. But in my situation application context is created but before consume and producing any message the context is closed, only the flush method of KafkaTemplate is called internally that force it to create a transactional producer but does not put it in the cache. As I didn't start a producer and KafkaTemplate do it on flush, is not it good that DefaultKafkaProducerFactory put them in cache before using them?
The producer cannot be closed if this template operation is participating in a transaction that was started outside of the template.
Even when closed, it is only "logically" closed - cached for reuse by another operation.
Is it a correct behaviour?
Yes, for producer-initiated transactions; the alternative name is used when a consumer initiates the transaction.
The InstanceAlreadyExistsException problem is simply because you are creating two application contexts with identical configuration. Why are you doing that?
Related
There is the following Spring Integration configuration with Retry:
<int:chain input-channel="sdCreationChannel" output-channel="debugLogger">
<int:poller fixed-delay="500" />
<int:filter ref="sdIntegrationExistingRequestSentFilter" method="filter"/>
<int:service-activator ref="sdCreationServiceImpl" method="processMessage">
<int:request-handler-advice-chain>
<ref bean="retryAdvice"/>
</int:request-handler-advice-chain>
</int:service-activator>
</int:chain>
<bean id="retryAdvice" class="org.springframework.integration.handler.advice.RequestHandlerRetryAdvice" >
<property name="retryTemplate">
<bean class="org.springframework.retry.support.RetryTemplate">
<property name="backOffPolicy">
<bean class="org.springframework.retry.backoff.ExponentialBackOffPolicy">
<property name="initialInterval" value="${integration.retry.initial.delay}"/>
<property name="multiplier" value="${integration.retry.backoff.multiplier}"/>
</bean>
</property>
<property name="retryPolicy">
<bean class="org.springframework.retry.policy.SimpleRetryPolicy">
<property name="maxAttempts" value="${integration.retry.max.attempts}" />
</bean>
</property>
</bean>
</property>
</bean>
The simplified Java code is as follows:
#Component("sdCreationServiceImpl")
public class SDCreationServiceImpl implements SDCreationService {
#Autowired
private NotifySD notifySD;
#Override
public void processMessage(IntegrationPayload integrationPayload) {
List<ConfirmationCode> sdConfCodes = findCodesFromPayLoad(integrationPayload);
notifySD.activateConfirmationCodes(sdConfCodes);
}
The problem with retrying this code is that the List sdConfCodes can be partially processed at every Retry, so every time we need to send for processing the less amount of elements. What is best way to organize this code?
Following the Artem Bilan suggestion (Thanks!) I created the 2nd method with variable list in SDCreationServiceImpl, i.e. activateConfirmationCodes and then in the XML specification pointed to this method as a method for sdCreationServiceImpl.
#Component("sdCreationServiceImpl")
public class SDCreationServiceImpl implements SDCreationService {
#Autowired
private NotifySD notifySD;
List<ConfirmationCode> sdConfCodes = new ArrayList<ConfirmationCode()>;
#Override
public void processMessage(IntegrationPayload integrationPayload) {
sdConfCodes = findCodesFromPayLoad(integrationPayload);
}
public void activateConfirmationCodes()
{
notifySD.activateConfirmationCodes(sdConfCodes);
}
And then the XML spec for service-activator is as follows:
<int:service-activator ref="sdCreationServiceImpl" method="activateConfirmationCodes">
<int:request-handler-advice-chain>
<ref bean="retryAdvice"/>
</int:request-handler-advice-chain>
</int:service-activator>
Yes, this method activateConfirmationCodes is invoked in Retry but the 1st method processMessage is not invoked at all. Is it possible to specify one method for invoking at the first try and other method for retrying?
Second with this design the list becomes the singleton and this can give problems in multithreading, correct?. Can this list be associated with a bean only for a particular message?
From big it isn't clear where is your problem. From other side let me share some my thoughts, perhaps I will guess your goal.
Having the List<ConfirmationCode> as a payload allows us to modify it at any time. So, let's assume we have list as 10 elements. On the first attempt we have processed 3 of them. The fourth has failed. We have to go to retry throwing some appropriate exception.
But we come back to the beginning of the retry-aware method, so with the same arguments. If we remove those successful items from the collection, the next retry iteration won't process them at all.
From one side you can achieve that distinguishing findCodesFromPayLoad() service and activateConfirmationCodes(), applying retry for the last one.
From other side you can mark items as processed in the activateConfirmationCodes(), so the next findCodesFromPayLoad(integrationPayload) won't return them.
In other words there are enough ways to modify collection without changing messages.
I'm trying to modify my spring-amqp project such that if a certain exception type is thrown, always requeue the message. Otherwise, retry x number of times then reject.
Here is the relevant configuration xml
<rabbit:connection-factory id="rabbitMqConnectionFactory" host="localhost" port="5672" />
<rabbit:template id="rabbitTemplate" connection-factory="rabbitMqConnectionFactory"
exchange="my.exchange" routing-key="foo"/>
<bean id="myListenerContainer" class="org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer">
<property name="connectionFactory" ref="rabbitMqConnectionFactory" />
<property name="messageListener" ref="myMessageListener" />
<property name="concurrentConsumers" value="2" />
<property name="queueNames" value="work.q" />
<property name="adviceChain" ref="myRetryInterceptor" />
</bean>
And here is the relevant code
#Bean(name="myRetryInterceptor")
public MethodInterceptor getInterceptor() {
return RetryInterceptorBuilder.stateless()
.retryPolicy(getRetryPolicy())
.recoverer(new RejectAndDontRequeueRecoverer())
.build();
}
private RetryPolicy getRetryPolicy() {
return new SimpleRetryPolicy(5, Collections.EMPTY_MAP) {
#Override
public boolean canRetry(RetryContext context) {
Throwable t = context.getLastThrowable();
if ((t!=null) && (t.getCause() instanceof com.test.MessageRetryException)) {
return true;
}
return super.canRetry(context);
}
};
}
I notice that when I deliberately throw a com.test.MessageRetryException in myMessageListener, the same thread is given the message over and over again. This is in contrast to an implementation where no spring classes are modified. In this case, the message is given alternately to one consumer thread then another. Am I doing something wrong?
#Bean(name="myRetryInterceptor")
//all consumers are given the message here
public MethodInterceptor getInterceptor() {
return RetryInterceptorBuilder.stateful()
.maxAttempts(5)
.recoverer(new RejectAndDontRequeueRecoverer())
.build();
}
No; you're doing nothing wrong; stateless retry is done within the scope of the original delivery (on the same thread). Message rejection does not occur until the retries are exhausted.
Stateful retry, on the other hand, rejects the delivery on each attempt and it will be resubmitted and (possibly) handled by a different thread.
Since rabbitmq now requeues rejected message at the head of the queue, there is not really any benefit of switching to stateful recovery and it requires a message id header to work (so the state can be determined for the message).
EDIT: Hmmm - I see you are using stateful retry; so I would expect the redeliveries to happen on alternate threads.
If you can post a DEBUG log on a Gist or some place; I can take a look. But, as I said, stateless retry is simpler anyway.
I am using Spring Integration (2.2.0) with WebSphere (8.0.0.x), in order to send messages via JMS (Tibco EMS).
Communication between components is working fine, but we have observed huge latencies between the messaging hops. These are in line with what we see in the EMS logs:
2014-09-30 06:04:19.940 [user#host]: Destroyed consumer (connid=19202559, sessid=28728543, consid=328585032) on queue 'test.queue3.request'
2014-09-30 06:04:19.969 [user#host]: Created consumer (connid=19202564, sessid=28728551, consid=328585054) on queue 'test.queue2.request'
2014-09-30 06:04:20.668 [user#host]: Destroyed consumer (connid=19202562, sessid=28728549, consid=328585048) on queue 'test.queue1.request'
2014-09-30 06:04:20.733 [user#host]: Created consumer (connid=19202567, sessid=28728555, consid=328585071) on queue 'test.queue5.request'
2014-09-30 06:04:20.850 [user#host]: Destroyed consumer (connid=19202563, sessid=28728550, consid=328585051) on queue 'test.queue4.request'
2014-09-30 06:04:21.001 [user#host]: Destroyed consumer (connid=19202564, sessid=28728551, consid=328585054) on queue 'test.queue2.request'
2014-09-30 06:04:21.701 [user#host]: Created consumer (connid=19202571, sessid=28728561, consid=328585093) on queue 'test.queue3.request'
2014-09-30 06:04:21.762 [user#host]: Destroyed consumer (connid=19202567, sessid=28728555, consid=328585071) on queue 'test.queue5.request'
Apparently, consumers are constantly being destroyed and re-created. This is not only bad for EMS, but also it's killing the latency, as messages are not delivered until the consumer is back online.
This is how the consumers are defined:
<jee:jndi-lookup id="rawConnectionFactory" jndi-name="jms/QueueCF"/>
<bean id="jmsDestinationResolver"
class="org.springframework.jms.support.destination.JndiDestinationResolver"/>
<bean id="connectionFactory"
class="org.springframework.jms.connection.UserCredentialsConnectionFactoryAdapter"
p:targetConnectionFactory-ref="rawConnectionFactory"
p:username="${jms.internal.username}"
p:password="${jms.internal.password}"/>
<bean id="taskExecutor"
class="org.springframework.scheduling.commonj.WorkManagerTaskExecutor"
p:workManagerName="wm/mc"
p:resourceRef="false"/>
<bean id="transactionManager"
class="org.springframework.transaction.jta.WebSphereUowTransactionManager"/>
<bean id="adp1Container"
class="org.springframework.jms.listener.DefaultMessageListenerContainer"
p:taskExecutor-ref="taskExecutor"
p:destinationName="requestQueue1" p:connectionFactory-ref="connectionFactory"
p:destinationResolver-ref="jmsDestinationResolver"
p:transactionManager-ref="transactionManager" />
<jms:inbound-gateway id="jmsInAdapter1"
request-channel="adapter1logic" container="adp1Container" />
<channel id="adapter1logic" />
Update:
This behaviour is related to the use of the transaction manager.
If we specify the connection to the EMS server directly in Spring (indicating there the host, port, user, password), consumers are still constantly recreated, but for some reason these recreations are not affecting the end-to-end latencies. Connections are apparently being managed better in Spring than in WAS.
How to configure WAS so that consumers trigger as quick as in Spring?
If, along with the previous change, I also remove the reference to the transaction manager in the DefaultMessageListenerContainer, consumers stop destroying and constructing altogether.
What could be the issue with WebSphere's transaction manager? Why are consumers destroying and constructing when WAS' transaction manager is in use? Is there any configuration that can be adjusted?
You should not see consumers being recycled like that, unless your listener is throwing an exception. Container consumers a long-lived by default. I suggest you turn on DEBUG (or even TRACE) logging for the container to figure out what's going on.
Suggesting to wrap your connection factory with CachingConnectionFactory decorator and configure the session caching strategy:
<bean id="cacheConnFactory"
class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="connectionFactory" />
<property name="cacheProducers" value="true" />
<property name="cacheConsumers" value="true" />
<property name="sessionCacheSize" value="10" />
</bean>
Use the above connection factory in your DMLC along with cacheLevel settings as follows:
<bean id="adp1Container"
class="org.springframework.jms.listener.DefaultMessageListenerContainer"
p:taskExecutor-ref="taskExecutor"
p:destinationName="requestQueue1"
p:connectionFactory-ref="cacheConnFactory"
p:destinationResolver-ref="jmsDestinationResolver"
p:transactionManager-ref="transactionManager">
<property name="sessionTransacted" value="true" />
<property name="cacheLevel" value="3" /> <!-- Consumer Level -->
</bean>
I'm using spring integration to invoke a service on the other end of an active mq. My config looks like:
<bean id="jmsConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<constructor-arg>
<bean class="org.apache.activemq.ActiveMQConnectionFactory"
p:brokerURL="${risk.approval.queue.broker}"
p:userName="${risk.approval.queue.username}"
p:password="${risk.approval.queue.password}"
/>
</constructor-arg>
<property name="reconnectOnException" value="true"/>
<property name="sessionCacheSize" value="100"/>
</bean>
<!-- create and close a connection to prepopulate the pool -->
<bean factory-bean="jmsConnectionFactory" factory-method="createConnection" class="javax.jms.Connection"
init-method="close" />
<integration:channel id="riskApprovalRequestChannel"/>
<integration:channel id="riskApprovalResponseChannel"/>
<jms:outbound-gateway id="riskApprovalServiceGateway"
request-destination-name="${risk.approval.queue.request}"
reply-destination-name="${risk.approval.queue.response}"
request-channel="riskApprovalRequestChannel"
reply-channel="riskApprovalResponseChannel"
connection-factory="jmsConnectionFactory"
receive-timeout="5000"/>
<integration:gateway id="riskApprovalService" service-interface="com.my.super.ServiceInterface"
default-request-channel="riskApprovalRequestChannel"
default-reply-channel="riskApprovalResponseChannel"/>
What I've noticed is that with this config the consumers created to grab the matching request from active mq never close. Every request increments the consumer count.
I can stop this from happening by adding
<property name="cacheConsumers" value="false" />
To the CachingConnectionFactory.
However according to the java docs for CachingConnectionFactory :
Note that durable subscribers will only be cached until logical
closing of the Session handle.
Which suggests that the session is never being closed.
Is this a bad thing? Is there a better way to stop the consumers from piling up?
Cheers,
Peter
First, you don't need the init-method on your factory-bean - it does nothing - the session factory only has one connection and calling close() on it is a no-op. (CCF is a subclass of SingleConnectionFactory).
Second; caching consumers is the default; sessions are never closed, unless the number of sessions exceeds the sessionCacheSize (which you have set to 100).
When close() is called on a cached session, it is cached for reuse; that's what the caching connection factory is for - avoiding the overhead of session creation for every request.
If you don't want the performance benefit of caching sessions, producers and consumers, use the SingleConnectionFactory instead. See the JavaDoc for CachingConnectionFactory.
Does the following work when using cachingConnectionFactory?
In your spring config file add in the connection factory config details something like this: cacheConsumers="false"
Default Behaviour is true which was causing a connection leak in the Queue.
I'm reading Spring Framework reference, chapter about JMS integration. There are some examples for sending text messages and asynchronously receiving them (by listeners). And there is also an example for JmsTemplate function convertAndSend which converts given object to a message. The reference says:
By using the converter, you and your application code can focus on the business object that is being sent or received via JMS and not be concerned with the details of how it is represented as a JMS message.
But there is no example for receiving such messages. They mention function receiveAndConvert but, unfortunately, it receives synchronously.
So how am I to receive it asynchronously? Must I be aware that when I convertAndSend a Map, the resulting message will be a MapMessage, and just check in my listener for this type of message and handle it? But they promised I'm not to be concerned with the details of how it is represented as a JMS message.
So is there a better way?
I know it's been a while since this was asked, but I had the same problem, solved it and wanted to give an explicit code example here.
Here's my MessageListener. This implements the onMessage(Message) method to intercept messages asynchronously.
package com.package.amqp;
import org.springframework.amqp.core.Message;
import org.springframework.amqp.core.MessageListener;
import org.springframework.amqp.support.converter.JsonMessageConverter;
import com.package.model.User;
public class TestListener implements MessageListener {
public void onMessage(Message message) {
JsonMessageConverter jmc = new JsonMessageConverter();
User u = (User)jmc.fromMessage(message);
System.out.println("received: " + u.getFirstName());
}
}
The messages are then converted using the standard JsonMessageConvertor in my case as this is the messageConvertor I plugged into my rabbitTemplate bean.
<bean id="rabbitConnectionFactory" class="org.springframework.amqp.rabbit.connection.SingleConnectionFactory">
<constructor-arg value="10.10.1.2"/>
<property name="username" value="guest"/>
<property name="password" value="guest"/>
</bean>
<bean class="org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer">
<property name="connectionFactory" ref="rabbitConnectionFactory"/>
<property name="queueName" value="queue.helloWorld"/>
<property name="messageListener" ref="someListener"/>
</bean>
<bean id="someListener" class="com.package.amqp.TestListener"></bean>
<bean id="rabbitTemplate" class="org.springframework.amqp.rabbit.core.RabbitTemplate">
<property name="connectionFactory" ref="rabbitConnectionFactory"/>
<property name="messageConverter">
<bean class="org.springframework.amqp.support.converter.JsonMessageConverter"/>
</property>
</bean>
Hope this helps someone!
Owen
While JmsTemplate provides basic synchronous receive methods, asynchronous reception is a whole lot more complicated, and is beyond the scope of JmsTemplate.
Asynchronous reception of JMS messages is done in Spring using Message Listener Containers, which asynchronously take messages from the JMS destination and pass them to your application. You can plug a MessageConverter in to your message listener container via a MessageListenerAdapter (plug the converter into the adapter, plug your application's listener into the adapter, then plug the adapter into the listener container).