I am new to RabbitMQ, I just went through Rabbitmq docs (Routing). I am quite confused between Exchange with routing keys. My requirement is , I want to create multiple queues dynamically. Please refer below diagram.
Ex. Lets say If producer create message for consumer c3, then it should go to Exchange and route to Queue 3 only and consume by C3 only.
At present I only require 3 Queues in future this count may increase. so how to deal with this situation too.
Note : I refer this blog Exchange
I have used Spring hibernate along with Rabbitmq.
below code shows Rabbit MQ Listener configuration File.
<?xml version="1.0" encoding="UTF-8" ?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:rabbit="http://www.springframework.org/schema/rabbit"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.1.xsd
http://www.springframework.org/schema/rabbit
http://www.springframework.org/schema/rabbit/spring-rabbit-1.0.xsd">
<rabbit:connection-factory id="connectionFactory" host="xx.xx.10.181" username="admin" password="admin" />
<rabbit:admin connection-factory="connectionFactory" />
<!-- Create Queues -->
<rabbit:queue id="q1" name="q1.queue" durable="true" />
<rabbit:queue id="q2" name="q2.queue" durable="true" />
<rabbit:queue id="q3" name="q3.queue" durable="true" />
<!--create Exchange here -->
<rabbit:direct-exchange id="myExchange" name="MY Exchange">
<rabbit:bindings>
<rabbit:binding queue="q1" key="my.queue.q1" />
<rabbit:binding queue="q2" key="my.queue.q2" />
<rabbit:binding queue="q3" key="my.queue.q3" />
</rabbit:bindings>
</rabbit:direct-exchange>
<!-- instantiate Listeners -->
<bean id="q1Listener" class="in.my.brocker.amqp.Q1Listener" />
<bean id="q2Listener" class="in.my.brocker.amqp.Q2Listener" />
<bean id="q3Listener" class="in.my.brocker.amqp.Q3Listener" />
<!-- glue the listener and myAnonymousQueue to the container-->
<rabbit:listener-container id="myListenerContainer" connection-factory="connectionFactory">
<rabbit:listener ref="q1Listener" queues="q1" />
<rabbit:listener ref="q2Listener" queues="q2" />
<rabbit:listener ref="q3Listener" queues="q3" />
</rabbit:listener-container>
</beans>
In above rabbit-listener-context.xml, I have created 3 Queues along with 3 Listener Classes. My aim is to Queue should be access by defined routing keys. I want to create nth no of queues this dynamically? How can we achieve this?
Related
Is it possible to send messages with SimpMessageSendingOperations from a RabbitMQ listener bean?
I have the following listener class:
public class MyJobListener {
#Autowired
public SimpMessageSendingOperations messagingTemplate;
public void handleJob(JobMessage jobMessage) {
doWork(jobMessage);
messagingTemplate.convertAndSend("/topic/greetings", "TEST");
}
}
My Rabbit config file is:
<!-- RabbitMQ configuration -->
<rabbit:connection-factory id="connectionFactory" host="${rabbitmq.connection.host}" port="${rabbitmq.connection.port}" />
<rabbit:admin connection-factory="connectionFactory" />
<rabbit:template id="amqpTemplate" connection-factory="connectionFactory" />
<!-- Queues -->
<rabbit:queue id="myQueue" name="myQueue" />
<!-- Listeners -->
<bean id="myListener01" class="com.xxx.MyJobListener" />
<bean id="myListener02" class="com.xxx.MyJobListener" />
<bean id="myListener03" class="com.xxx.MyJobListener" />
<bean id="myListener04" class="com.xxx.MyJobListener" />
<rabbit:listener-container connection-factory="connectionFactory" >
<rabbit:listener ref="myListener01" method="handleJob" queue-names="myQueue" />
<rabbit:listener ref="myListener02" method="handleJob" queue-names="myQueue" />
<rabbit:listener ref="myListener03" method="handleJob" queue-names="myQueue" />
<rabbit:listener ref="myListener04" method="handleJob" queue-names="myQueue" />
</rabbit:listener-container>
<!-- Bindings -->
<rabbit:direct-exchange name="directexchange" >
<rabbit:bindings>
<rabbit:binding queue="myQueue"/>
</rabbit:bindings>
</rabbit:direct-exchange>
When message is expected to be sent (messagingTemplate.convertAndSend("/topic/greetings", "TEST")) nothing happens, but if I do the same thing but in a #Controller everything works fine (message is sent through websocket to the browser)
I need to do this to send a notification to the user when the job is finished.
After many tests I changed my rabbit configuration file, leaving only one listener:
<!-- Listeners -->
<bean id="myListener01" class="com.xxx.MyJobListener" />
<rabbit:listener-container connection-factory="connectionFactory" error-handler="queueErrorHandler" >
<rabbit:listener ref="myListener01" method="handleJob" queue-names="myQueue" />
</rabbit:listener-container>
and now it works almost randomly. It's strange, but each 2 calls it works. I mean, two times yes, two times not, two times yes, two times not... and so... It's very strange. I think there is something with the rabbit config...
Definitely is Spring Security configuration. If I disable Spring Security everything works fine. I will find out what it is, and then I'll post the answer here.
I was able to solve it.
The problem was not Spring Security, the problem was I was declarating twice the websocket message broker:
<websocket:message-broker application-destination-prefix="/app" >
<websocket:stomp-endpoint path="/websocket" >
<websocket:sockjs />
</websocket:stomp-endpoint>
<websocket:simple-broker prefix="/topic,/user" />
</websocket:message-broker>
These lines resides in my websocket.xml, and this file was imported more than one time because of an "ugly" import sentences distributions along my .xml spring files.
After ordering these imports and ensuring the bean is only created once everything works fine.
May this helps!
I'm using Spring AMQP to listen to messages (configuration has listener-container, service-activator, chain, bridge & aggregators). At application startup AMQP starts reading messages which we don't want. I tried auto-startup=false but it isn't working. Am I missing anything?
Also, if it does work then how do I programmatically start them again? I tried listenerContainer.start();. What about aggregators & others?
EDIT
Following is my config:
<rabbit:queue name="my_queue1" declared-by="consumerAdmin"/>
<rabbit:queue name="my_queue2" declared-by="consumerAdmin"/>
<rabbit:queue name="my_batch1" declared-by="consumerAdmin"/>
<int-amqp:channel id="myPollableChannel" message-driven="false" connection-factory="consumerConnFactory" queue-name="my_queue2"/>
<int-event:inbound-channel-adapter channel="myPollableChannel" auto-startup="false"/>
<int-amqp:channel id="myAggregateChannel" connection-factory="consumerConnFactory"/>
<int-event:inbound-channel-adapter channel="myAggregateChannel" auto-startup="false"/>
<int-amqp:channel id="myChannel" connection-factory="consumerConnFactory"/>
<int-event:inbound-channel-adapter channel="myChannel" auto-startup="false"/>
<int-amqp:channel id="myFailedChannel" connection-factory="consumerConnFactory"/>
<int-event:inbound-channel-adapter channel="myFailedChannel" auto-startup="false"/>
<rabbit:template id="genericTopicTemplateWithRetry" connection-factory="connectionFactory" exchange="my_exchange" retry-template="retryTemplate"/>
<rabbit:topic-exchange name="my_exchange" declared-by="consumerAdmin">
<rabbit:bindings>
<rabbit:binding queue="my_queue1" pattern="pattern1"/>
<rabbit:binding queue="my_queue2" pattern="pattern1"/>
</rabbit:bindings>
</rabbit:topic-exchange>
<int:handler-retry-advice id="retryAdvice" max-attempts="5" recovery-channel="myFailedChannel">
<int:exponential-back-off initial="3000" multiplier="5.0" maximum="300000"/>
</int:handler-retry-advice>
<int:bridge input-channel="myPollableChannel" output-channel="myAggregateChannel">
<int:poller max-messages-per-poll="100" fixed-rate="5000"/>
</int:bridge>
<int:aggregator id="myBatchAggregator"
ref="myAggregator"
correlation-strategy="myCorrelationStrategy"
release-strategy="myReleaseStrategy"
input-channel="myAggregateChannel"
output-channel="myChannel"
expire-groups-upon-completion="true"
send-partial-result-on-expiry="true"
group-timeout="1000" />
<int:chain input-channel="myFailedChannel">
<int:transformer expression="'Failed to publish messages to my channel:' + payload.failedMessage.payload" />
<int-stream:stderr-channel-adapter append-newline="true"/>
</int:chain>
<int:service-activator input-channel="myChannel" output-channel="nullChannel" ref="myWorker" method="myMethod">
<int:request-handler-advice-chain><ref bean="retryAdvice" /></int:request-handler-advice-chain>
</int:service-activator>
<rabbit:listener-container connection-factory="consumerConnFactory" requeue-rejected="false" concurrency="1">
<rabbit:listener ref="myListener" method="listen" queue-names="queues1" admin="consumerAdmin" />
</rabbit:listener-container>
OK. Thank you for the configuration!
Not sure why you need AMQP-bascked channels, but the main issue for you is exactly there.
But pay attention, the <int-amqp:channel> has auto-startup="false" option as well.
And you will be ready to receive data from the AMQP, you will need just start() those channels by their id.
I'm used to camel, where it is somewhat simple to pipeline output from one element to the input of another.
I want to send all application events to an AMQP queue, the fire hose, and then route events to different queues depending on the event type. For example, if the event type is session.created I'd like to take it out from the fire hose and send it to the session.created queue.
I have defined the following raabitmq configuration
<rabbit:connection-factory id="connectionFactory" host="localhost"/>
<rabbit:admin connection-factory="connectionFactory"/>
<rabbit:template id="amqpTemplate" connection-factory="connectionFactory"/>
<rabbit:queue name="q.firehose"/>
<rabbit:queue name="q.session.created"/>
<rabbit:direct-exchange name="e.firehose">
<rabbit:bindings>
<rabbit:binding key="firehose" queue="q.firehose"/>
</rabbit:bindings>
</rabbit:direct-exchange>
<rabbit:headers-exchange name="e.router">
<rabbit:bindings>
<rabbit:binding queue="q.session.created">
<rabbit:binding-arguments>
<entry key="x-match" value="all"/>
<entry key="event_type" value="session.created"/>
</rabbit:binding-arguments>
</rabbit:binding>
</rabbit:bindings>
</rabbit:headers-exchange>
And I want to try something like this spring integration configuration:
<int:channel id="fromFirehose"/>
<int:channel id="toRouter"/>
<int-amqp:inbound-channel-adapter channel="fromFirehose" queue-names="q.firehose" connection-factory="connectionFactory"/>
<!-- Some config element here to move all input from the firehose out and put it into e.router--/>
<int-amqp:outbound-channel-adapter channel="toRouter" exchange-name="e.router" amqp-template="routerTemplate" />
Which component is best suited to move input from the fire hose into the e.router exchange? Is this a good approach?
Looks like a transformer can move from messages from one channel to the other but you are obliged to apply a transformation. If there is no other way, is it there a DoNothingTransformer available?
Thanks in advance!
Just link both inbound and outbound adapters to the same channel.
For example:
<int:channel id="fromFirehose"/>
<int-amqp:inbound-channel-adapter channel="fromFirehose" queue-names="q.firehose" connection-factory="connectionFactory"/>
<int-amqp:outbound-channel-adapter channel="fromFirehose" exchange-name="e.router" amqp-template="routerTemplate" />
I have 2 spring integration context files using similar file-based integration pattern. Both scan directory looking for a message and both work if deployed by themselves. If I include both modules into another spring context they are loading without issues. However only second one is working, and the first one is getting: MessageDeliveryException: Dispatcher has no subscribers. I've attempted to combine them into a single context file with no positive gain. We are currently on version 2.1.3 of Spring Integration and version 2.1 of Spring Integration File. Any ideas are greatly appreciated!
inpayment-context.xml:
<!-- START of in-bound message implementation -->
<int:channel id="file-inpayment-channel" datatype="java.io.File" />
<bean id="xmlPatternFileListFilter" class="org.springframework.integration.file.filters.SimplePatternFileListFilter">
<constructor-arg value="*.xml" />
</bean>
<task:executor id="batchInBoundExecuter" pool-size="1-1" queue-capacity="20" rejection-policy="CALLER_RUNS" />
<int-file:inbound-channel-adapter directory="file:${inpayment.inbox}" filter="xmlPatternFileListFilter"
channel="file-inpayment-channel">
<int:poller id="inPaymentrPoller" fixed-delay="1000" task-executor="batchInBoundExecuter" default="true" />
</int-file:inbound-channel-adapter>
<bean id="inPaymentService" class="com.somepackage.InPaymentBootstrapService" />
<int:service-activator id="batchJobLaunchService" ref="inPaymentService" input-channel="file-inpayment-channel"
method="schedule" />
<!-- START of out-bound message implementation -->
<int:channel id="inpayment-file-out-channel" datatype="java.io.File" />
<int:gateway id="inboundPaymentGateway" service-interface="com.somepackage.InboundPaymentGateway"
default-request-channel="inpayment-file-out-channel" />
<int-file:outbound-channel-adapter directory="file:${inpayment.inprocess}" channel="inpayment-file-out-channel"
auto-create-directory="true" delete-source-files="true" />
<!-- END of out-bound message implementation -->
scheduler-context.xml:
<!-- START of in-bound message implementation -->
<int:channel id="scheduler-file-in-channel" datatype="java.io.File" />
<bean id="simplePatternFileListFilter" class="org.springframework.integration.file.filters.SimplePatternFileListFilter">
<constructor-arg value="*.xml" />
</bean>
<task:executor id="batchJobRunExecuter" pool-size="1-1" queue-capacity="20" rejection-policy="CALLER_RUNS"/>
<int-file:inbound-channel-adapter directory="file:${scheduler.inbox}" filter="simplePatternFileListFilter"
channel="scheduler-file-in-channel">
<int:poller id="schedulerPoller" fixed-delay="5000" task-executor="batchJobRunExecuter" default="true" />
</int-file:inbound-channel-adapter>
<bean id="launchService" class="com.somepackage.BatchJobLaunchService" />
<int:service-activator id="batchJobLaunchService" ref="launchService" input-channel="scheduler-file-in-channel"
method="schedule" />
<!-- END of in-bound message implementation -->
<!-- START of out-bound message implementation -->
<int:channel id="scheduler-file-out-channel" datatype="java.io.File" />
<int:channel id="scheduler-xml-out-channel" datatype="com.somepackage.ScheduledJob" />
<int:gateway id="batchJobSchedulerGateway" service-interface="com.innovation.customers.guideone.scheduler.integration.SchedulerGateway"
default-request-channel="scheduler-xml-out-channel" />
<int:transformer input-channel="scheduler-xml-out-channel" output-channel="scheduler-file-out-channel" ref="schedulerFileTransformer"
method="transformToFile" />
<int-file:outbound-channel-adapter directory="file:${scheduler.completed}" channel="scheduler-file-out-channel"
auto-create-directory="true" delete-source-files="true" />
<!-- END of out-bound message implementation -->
Common Spring Context:
<context:component-scan base-package="com.somepackage" />
<import resource="classpath:g1-scheduler-context.xml"/>
<import resource="classpath:g1-inpayment-context.xml"/>
EDIT
2014-08-27 11:01:01,530 ERROR [batchJobRunExecuter-1][:] org.springframework.integration.handler.LoggingHandler : org.springframework.integration.MessageDeliveryException: Dispatcher has no subscribers. at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:108) at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:101)
I see the issue in your config:
<int:service-activator id="batchJobLaunchService" ref="inPaymentService" input-channel="file-inpayment-channel"
method="schedule" />
and
<int:service-activator id="batchJobLaunchService" ref="launchService" input-channel="scheduler-file-in-channel"
method="schedule" />
They assume to be different services, but at the same time use the same id - batchJobLaunchService.
Spring by default allows to do that, but only the last bean definition with the same id wins. That's why the <service-activator> for the launchService hasn't been pupolated and hence the EventDrivenConsumer bean hasn't been subscribed to the scheduler-file-in-channel.
Be careful and use unique id for all your beans.
It isn't so easy to throw expection on the duplication case, but if you switch on INFO for the org.springframework category you'll the message that one bean overrrides another.
I hava following spring xml configuration
<bean id="connectionFactory"
class="org.springframework.amqp.rabbit.connection.CachingConnectionFactory">
<constructor-arg value="xxxxxxxx"/>
<property name="username" value="xxxxx"/>
<property name="password" value="xxxxx"/>
<property name="channelCacheSize" value="25"/>
<property name="virtualHost" value="/"/>
<property name="port" value="3453"/>
</bean>
<rabbit:template id="tutorialTemplate" connection-factory="connectionFactory"/>
<!-- 1st queue -->
<rabbit:queue id="veliteQueue" name="ES_queue" durable="true" auto-delete="false" exclusive="false"/>
<rabbit:direct-exchange id="myExchange" durable="true" name="ES_exchange">
<rabbit:bindings>
<rabbit:binding queue="veliteQueue" key="logstash"></rabbit:binding>
</rabbit:bindings>
</rabbit:direct-exchange>
<!-- 2nd Queue -->
<rabbit:queue id="veliteQueue1" name="ES_queue_Temp" durable="true" auto-delete="false" exclusive="false"/>
<rabbit:direct-exchange id="myExchange1" durable="true" name="ES_exchange_temp">
<rabbit:bindings>
<rabbit:binding queue="ES_queue_Temp" key="logstash_temp"></rabbit:binding>
</rabbit:bindings>
</rabbit:direct-exchange>
<!-- 2 Listeners for 2 queue's mentioned above -->
<bean id="aListener" class="com.vzw.es.cosumer.SpringMessageListener" autowire="byName"/>
<bean id="aListener1" class="com.vzw.es.cosumer.SpringMessageListener1" autowire="byName"/>
<rabbit:listener-container id="myListenerContainer" connection-factory="connectionFactory" acknowledge="auto" prefetch="750" concurrency="1">
<rabbit:listener ref="aListener" queues="veliteQueue"/>
<rabbit:listener ref="aListener1" queues="veliteQueue1"/>
</rabbit:listener-container>
Now in my Java code I have 2 Listener classes: com.vzw.es.cosumer.SpringMessageListener and com.vzw.es.cosumer.SpringMessageListener1. Now When I am running my main class only 1 listener's onMessage method is getting invoked i.e. SpringMessageListener1, I did check from RabbitMQ prespective and bothe queues have enough messages to consume.
Also when I comment out the 2nd queue and its listener from xml SpringMessageListener works perfectly.
It's a bug in the container parser, each listener gets its own container (the namespace is just a convenient way to specify common attributes). If you remove the id="myListenerContainer", it will work - because each container gets a (different) generated name. With the id specified, both beans get the same name, and the last definition replaces the first.
Alternatively, declare two separate container elements, with different IDs, and each having just one listener.
Thanks for finding this.
Please open a JIRA issue
EDIT: This issue was resolved in version 1.2.1.