I'm fairly new to Spring JMS, and I've found lots of documentation and examples at the Spring site and elsewhere, but my use case doesn't seem to be described anywhere, or at least in a way I can understand. I hope you might be able to help.
I would like to create a publisher of a topic and several durable subscribers to that topic. I'm working on the first subscriber now, and it is intended to run hourly (on a timer) and drain the topic of messages and process them all at once (i.e. to send an email summarizing all messages).
I do not know how to configure this setup in Spring, although I feel like this should be easy. Advice would be tremendously helpful.
My plan, such as it is, was to have the timer invoke the "processBatch" method, which would call receiveAndConvert() in a loop until it timed out, building up its list of messages.
This doesn't seem to work, though, because the consumer isn't really subscribed to the topic. Certainly not before it's run, and potentially not afterward.
How can I configure this using Spring and/or direct ActiveMQ?
I'm not sure where I ended up with my XML is a useful place for this discussion to start, but I'll provide it in case it is helpful:
<beans>
<!-- some unrelated beans -->
<!-- my Active MQ connection factory -->
<bean id="mqConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" destroy-method="stop">
<property name="connectionFactory">
<bean class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="vm://broker"/>
</bean>
</property>
</bean>
<!-- my topic -->
<amq:topic id="completionsTopic" physicalName="completions.topic"/>
<!-- my subscriber -->
<bean id="emailer" class="com.j128.Emailer">
<property name="jmsTemplate">
<bean class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="mqConnectionFactory"/>
<property name="defaultDestination" ref="completionsTopic"/>
<property name="receiveTimeout" value="2000"/>
</bean>
</property>
</bean>
<!-- my scheduler and periodic call to the topic drainer -->
<task:scheduler id="taskScheduler" pool-size="10"/>
<task:scheduled-tasks>
<!-- send emails hourly -->
<task:scheduled ref="emailer" method="processBatch" cron="0 * * * *"/>
</task:scheduled-tasks>
</beans>
But I'm certain I fundamentally have the wrong strategy and that there's a simple way to configure this.
Thank you for your assistance.
Have a look at how-does-a-queue-compare-to-a-topic: "Only subscribers who had an active subscription at the time the broker receives the message will get a copy of the message."
And for a durable topic how-do-durable-queues-and-topics-work: "Durable topics however are different as they must logically persist an instance of each suitable message for every durable consumer - since each durable consumer gets their own copy of the message"
So with a non-durable topic, your plan wont work as the hourly job won't get any messages as it isn't running when the messages are published. If you set up a durable topic then it might work but it depends what you expect to happen when you say your subscriber will "drain the topic of messages". All it can do it read the messages published to it since it last run, it can't affect the messages going to other subscribers.
For a discussion around durable subscribers (I haven't used them on ActiveMQ) see this
Related
I am using apache active mq, and have a common consumer for multiple operations. For every consumer it will take around 15-20 mins for code execution, and I want if in between the another message is produced then both messages should be consumed in parallel though both have one consumer only.
In application context where you have defined the DefaultMessageListenerContainer bean, You can add "concurrentConsumers" property and define the number of concurrent consumers you want, to listen to your message Queue.
Below is an example for the same
<bean id="jmsMailContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="concurrentConsumers">
<value>100</value>
</property>
<property name="connectionFactory" ref="connectionFactory"/>
<property name="destination" ref="mailDestination"/>
<property name="messageListener" ref="jmsMailConsumer"/>
</bean>
I have a problem with database transactions in mule flow. This is the flow that i have defined:
<flow name="createPortinCaseServiceFlow">
<vm:inbound-endpoint path="createPortinCase" exchange-pattern="request-response">
<custom-transaction action="ALWAYS_BEGIN" factory-ref="muleTransactionFactory"/>
</vm:inbound-endpoint>
<component>
<spring-object bean="checkIfExists"/>
</component>
<component>
<spring-object bean="createNewOne"/>
</component>
</flow>
The idea is that in checkIfExists we verify if some data exists (in the database) if it does we throw an exception. If it does not we go to createNewOne and create a new data.
The problem
is that if we run the flow concurrently new objects will be created multiple times in createNewOne and they should not be as we invoke checkIfExists just before it. This means that the transaction is not working properly.
More info:
both createNewOne and checkIfExists have following annotation:
#Transactional(propagation = Propagation.MANDATORY)
Definition of muleTransactionFactory looks as follows
<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="dataSource" ref="teleportNpDataSource"/>
<property name="entityManagerFactory" ref="npEntityManagerFactory"/>
<property name="nestedTransactionAllowed" value="true"/>
<property name="defaultTimeout" value="${teleport.np.tm.transactionTimeout}"/>
</bean>
<bean id="muleTransactionFactory" class="org.mule.module.spring.transaction.SpringTransactionFactory">
<property name="manager" ref="transactionManager"/>
</bean>
I have set the TRACE log level (as #Shailendra suggested) and i have discovered that transaction is reused in all of the spring beans:
00:26:32.751 [pool-75-thread-1] DEBUG org.springframework.orm.jpa.JpaTransactionManager - Participating in existing transaction
In the logs transaction is commited at the same time which means that those transactions are created properly but there are executed concurrently which causes an issue.
Issue may be because multi-threading. When you post multiple objects to VM, they will be dispatched to multiple receiving threads and if multi-threading is not properly handled in your components then you might run into the issue you mentioned.
Test making these change - Add VM connector reference and turn off the dispatcher threading profile. That way, VM will process messages one at a time as there is just one dispatcher thread.
<vm:connector name="VM" validateConnections="true" doc:name="VM" >
<dispatcher-threading-profile doThreading="false"/>
</vm:connector>
<flow name="testFlow8">
<vm:inbound-endpoint exchange-pattern="one-way" doc:name="VM" connector-ref="VM">
<custom-transaction action="NONE"/>
</vm:inbound-endpoint>
</flow>
Be aware of the fact that, if the number of incoming messages on VM are very high and time taken to process each message is more, then you may run into SEDA-QUEUE errors due to no thread availability.
If without threading, your flow is behaving correctly then you may need to look how your components should behave in multi-threading.
Hope That helps!
I am using Spring Integration (2.2.0) with WebSphere (8.0.0.x), in order to send messages via JMS (Tibco EMS).
Communication between components is working fine, but we have observed huge latencies between the messaging hops. These are in line with what we see in the EMS logs:
2014-09-30 06:04:19.940 [user#host]: Destroyed consumer (connid=19202559, sessid=28728543, consid=328585032) on queue 'test.queue3.request'
2014-09-30 06:04:19.969 [user#host]: Created consumer (connid=19202564, sessid=28728551, consid=328585054) on queue 'test.queue2.request'
2014-09-30 06:04:20.668 [user#host]: Destroyed consumer (connid=19202562, sessid=28728549, consid=328585048) on queue 'test.queue1.request'
2014-09-30 06:04:20.733 [user#host]: Created consumer (connid=19202567, sessid=28728555, consid=328585071) on queue 'test.queue5.request'
2014-09-30 06:04:20.850 [user#host]: Destroyed consumer (connid=19202563, sessid=28728550, consid=328585051) on queue 'test.queue4.request'
2014-09-30 06:04:21.001 [user#host]: Destroyed consumer (connid=19202564, sessid=28728551, consid=328585054) on queue 'test.queue2.request'
2014-09-30 06:04:21.701 [user#host]: Created consumer (connid=19202571, sessid=28728561, consid=328585093) on queue 'test.queue3.request'
2014-09-30 06:04:21.762 [user#host]: Destroyed consumer (connid=19202567, sessid=28728555, consid=328585071) on queue 'test.queue5.request'
Apparently, consumers are constantly being destroyed and re-created. This is not only bad for EMS, but also it's killing the latency, as messages are not delivered until the consumer is back online.
This is how the consumers are defined:
<jee:jndi-lookup id="rawConnectionFactory" jndi-name="jms/QueueCF"/>
<bean id="jmsDestinationResolver"
class="org.springframework.jms.support.destination.JndiDestinationResolver"/>
<bean id="connectionFactory"
class="org.springframework.jms.connection.UserCredentialsConnectionFactoryAdapter"
p:targetConnectionFactory-ref="rawConnectionFactory"
p:username="${jms.internal.username}"
p:password="${jms.internal.password}"/>
<bean id="taskExecutor"
class="org.springframework.scheduling.commonj.WorkManagerTaskExecutor"
p:workManagerName="wm/mc"
p:resourceRef="false"/>
<bean id="transactionManager"
class="org.springframework.transaction.jta.WebSphereUowTransactionManager"/>
<bean id="adp1Container"
class="org.springframework.jms.listener.DefaultMessageListenerContainer"
p:taskExecutor-ref="taskExecutor"
p:destinationName="requestQueue1" p:connectionFactory-ref="connectionFactory"
p:destinationResolver-ref="jmsDestinationResolver"
p:transactionManager-ref="transactionManager" />
<jms:inbound-gateway id="jmsInAdapter1"
request-channel="adapter1logic" container="adp1Container" />
<channel id="adapter1logic" />
Update:
This behaviour is related to the use of the transaction manager.
If we specify the connection to the EMS server directly in Spring (indicating there the host, port, user, password), consumers are still constantly recreated, but for some reason these recreations are not affecting the end-to-end latencies. Connections are apparently being managed better in Spring than in WAS.
How to configure WAS so that consumers trigger as quick as in Spring?
If, along with the previous change, I also remove the reference to the transaction manager in the DefaultMessageListenerContainer, consumers stop destroying and constructing altogether.
What could be the issue with WebSphere's transaction manager? Why are consumers destroying and constructing when WAS' transaction manager is in use? Is there any configuration that can be adjusted?
You should not see consumers being recycled like that, unless your listener is throwing an exception. Container consumers a long-lived by default. I suggest you turn on DEBUG (or even TRACE) logging for the container to figure out what's going on.
Suggesting to wrap your connection factory with CachingConnectionFactory decorator and configure the session caching strategy:
<bean id="cacheConnFactory"
class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="connectionFactory" />
<property name="cacheProducers" value="true" />
<property name="cacheConsumers" value="true" />
<property name="sessionCacheSize" value="10" />
</bean>
Use the above connection factory in your DMLC along with cacheLevel settings as follows:
<bean id="adp1Container"
class="org.springframework.jms.listener.DefaultMessageListenerContainer"
p:taskExecutor-ref="taskExecutor"
p:destinationName="requestQueue1"
p:connectionFactory-ref="cacheConnFactory"
p:destinationResolver-ref="jmsDestinationResolver"
p:transactionManager-ref="transactionManager">
<property name="sessionTransacted" value="true" />
<property name="cacheLevel" value="3" /> <!-- Consumer Level -->
</bean>
I'm using spring integration to invoke a service on the other end of an active mq. My config looks like:
<bean id="jmsConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<constructor-arg>
<bean class="org.apache.activemq.ActiveMQConnectionFactory"
p:brokerURL="${risk.approval.queue.broker}"
p:userName="${risk.approval.queue.username}"
p:password="${risk.approval.queue.password}"
/>
</constructor-arg>
<property name="reconnectOnException" value="true"/>
<property name="sessionCacheSize" value="100"/>
</bean>
<!-- create and close a connection to prepopulate the pool -->
<bean factory-bean="jmsConnectionFactory" factory-method="createConnection" class="javax.jms.Connection"
init-method="close" />
<integration:channel id="riskApprovalRequestChannel"/>
<integration:channel id="riskApprovalResponseChannel"/>
<jms:outbound-gateway id="riskApprovalServiceGateway"
request-destination-name="${risk.approval.queue.request}"
reply-destination-name="${risk.approval.queue.response}"
request-channel="riskApprovalRequestChannel"
reply-channel="riskApprovalResponseChannel"
connection-factory="jmsConnectionFactory"
receive-timeout="5000"/>
<integration:gateway id="riskApprovalService" service-interface="com.my.super.ServiceInterface"
default-request-channel="riskApprovalRequestChannel"
default-reply-channel="riskApprovalResponseChannel"/>
What I've noticed is that with this config the consumers created to grab the matching request from active mq never close. Every request increments the consumer count.
I can stop this from happening by adding
<property name="cacheConsumers" value="false" />
To the CachingConnectionFactory.
However according to the java docs for CachingConnectionFactory :
Note that durable subscribers will only be cached until logical
closing of the Session handle.
Which suggests that the session is never being closed.
Is this a bad thing? Is there a better way to stop the consumers from piling up?
Cheers,
Peter
First, you don't need the init-method on your factory-bean - it does nothing - the session factory only has one connection and calling close() on it is a no-op. (CCF is a subclass of SingleConnectionFactory).
Second; caching consumers is the default; sessions are never closed, unless the number of sessions exceeds the sessionCacheSize (which you have set to 100).
When close() is called on a cached session, it is cached for reuse; that's what the caching connection factory is for - avoiding the overhead of session creation for every request.
If you don't want the performance benefit of caching sessions, producers and consumers, use the SingleConnectionFactory instead. See the JavaDoc for CachingConnectionFactory.
Does the following work when using cachingConnectionFactory?
In your spring config file add in the connection factory config details something like this: cacheConsumers="false"
Default Behaviour is true which was causing a connection leak in the Queue.
I'm reading Spring Framework reference, chapter about JMS integration. There are some examples for sending text messages and asynchronously receiving them (by listeners). And there is also an example for JmsTemplate function convertAndSend which converts given object to a message. The reference says:
By using the converter, you and your application code can focus on the business object that is being sent or received via JMS and not be concerned with the details of how it is represented as a JMS message.
But there is no example for receiving such messages. They mention function receiveAndConvert but, unfortunately, it receives synchronously.
So how am I to receive it asynchronously? Must I be aware that when I convertAndSend a Map, the resulting message will be a MapMessage, and just check in my listener for this type of message and handle it? But they promised I'm not to be concerned with the details of how it is represented as a JMS message.
So is there a better way?
I know it's been a while since this was asked, but I had the same problem, solved it and wanted to give an explicit code example here.
Here's my MessageListener. This implements the onMessage(Message) method to intercept messages asynchronously.
package com.package.amqp;
import org.springframework.amqp.core.Message;
import org.springframework.amqp.core.MessageListener;
import org.springframework.amqp.support.converter.JsonMessageConverter;
import com.package.model.User;
public class TestListener implements MessageListener {
public void onMessage(Message message) {
JsonMessageConverter jmc = new JsonMessageConverter();
User u = (User)jmc.fromMessage(message);
System.out.println("received: " + u.getFirstName());
}
}
The messages are then converted using the standard JsonMessageConvertor in my case as this is the messageConvertor I plugged into my rabbitTemplate bean.
<bean id="rabbitConnectionFactory" class="org.springframework.amqp.rabbit.connection.SingleConnectionFactory">
<constructor-arg value="10.10.1.2"/>
<property name="username" value="guest"/>
<property name="password" value="guest"/>
</bean>
<bean class="org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer">
<property name="connectionFactory" ref="rabbitConnectionFactory"/>
<property name="queueName" value="queue.helloWorld"/>
<property name="messageListener" ref="someListener"/>
</bean>
<bean id="someListener" class="com.package.amqp.TestListener"></bean>
<bean id="rabbitTemplate" class="org.springframework.amqp.rabbit.core.RabbitTemplate">
<property name="connectionFactory" ref="rabbitConnectionFactory"/>
<property name="messageConverter">
<bean class="org.springframework.amqp.support.converter.JsonMessageConverter"/>
</property>
</bean>
Hope this helps someone!
Owen
While JmsTemplate provides basic synchronous receive methods, asynchronous reception is a whole lot more complicated, and is beyond the scope of JmsTemplate.
Asynchronous reception of JMS messages is done in Spring using Message Listener Containers, which asynchronously take messages from the JMS destination and pass them to your application. You can plug a MessageConverter in to your message listener container via a MessageListenerAdapter (plug the converter into the adapter, plug your application's listener into the adapter, then plug the adapter into the listener container).