ActiveMQ Message Groups are a wonderful feature for load balancing across multiple consumers. In short: a stream of messages is partitioned across multiple consumers of a single queue according to a group identifier embedded in the message (JMSXGroupID). (So, consumer 1 will get all messages with JMSXGroupID = a, consumer 2 will get all messages with JMSXGroupID = b, and so on.)
Now, imagine you have 2 queues: A and B, and imagine that a consistent taxonomy of JMSXGroupIDs is used in messages flowing through both queues. Will the consumer the broker chooses for JMSXGroupID = ABC on queue A be the consumer from the same connection that the broker chooses for JMSXGroupID = ABC on queue B?
I suspect the answer to the question as I've asked it is "no." There are too many variables in play: What happens if the consumer the broker chooses for A has no corresponding consumer for B? What happens if the consumer the broker chooses for A has multiple corresponding consumers for B? There's no obvious right answer in these cases.
However, can we simulate this behavior? For example, a consumer on a composite destination could be a viable solution -- make sure all consumers on A and B consume on the composite destination A,B and you might be in business -- but ActiveMQ does not appear to support consuming from composite destinations.
The only solution I've found is simply to push messages for both A and B on one single queue -- call it AB -- and have an exclusive consumer on that. You now have to distinguish between "A messages" and "B messages," but you could easily do that with headers.
However, this solution smells funny. (You now have to assume producers will dutifully apply special headers to their messages, or modify your payloads.) Is there a solution that will make sure consumers across two separate queues A and B always land on the same connection?
As you have correctly worked out, message groups apply to a single queue only. There is no coordination between multiple queues.
Generally when you are using message groups you are trying to guarantee message ordering across not just delivery, but processing - so that say all events for a particular entity are processed in sequence. The devil is always in the details of your use case, but putting all of the related messages onto a single queue will give you the result that you are after. In order to process them differently, you then need to put some sort of multiplexing logic into your consumer to make the decision based on the message payload - a well known header as you say is a good candidate for a solution.
To get around the prerequisite of ensuring that clients explicitly set this, what you can do is write a piece of Camel routing logic that does this on your behalf - this is only possible with the broker: component that was added to ActiveMQ 5.9. The idea will be that producers see two separate queues - A & B; the routing logic will read from those queues as the messages are being put in, set the header appropriately, and re-route them to C instead. The routing logic in effect works as an interceptor.
<route id="ConflateA">
<from uri="broker:queue:A"/>
<setHeader headerName="OriginalMessageSource">
<constant>A</constant>
</setHeader>
<to uri="broker:queue:C"/>
</route>
You can then use the OriginalMessageSource header in your multiplexing logic.
Related
I have a question about RabbitMQ queue. I would like to send two types of messages on just one queue.
I know that, I can create two different queues, and use routing key to send different messages to different queue.
But I would like to have two consumers on one Queue, and somehow bind Consumer with type of message. It's event's driven via rabbit queue, when client and core are publishers and consumers.
Is it possible, or should I use different Queues?
Data exchange
Like #kendavidson said, there is a possibility to use only one queue, to exchange different messages, but it is terrible idea, because it's not efficient, so you should use it only if it's truly nesesery.
I found comment #Петр Александров useful and I created separate queue for every consumer to fix my problem, and it's something you probably looking for.
I have two instances of my app subscribing to a topic. Since there are two instances(i.e two subscribers), two events(messages) will be generated and written to a queue. (Now i have duplicate message in the queue and one by one each is going to be processed) But I want to have a solution where only one event is processed/or only one message is written to the queue. How can i achieve that?. I must have two subscribers instead one goes down
JMS topics follow publish-subscribe semantics where every subscriber will get the message. However, JMS queues follow point-to-point semantics where only 1 of the connected consumers will receive the message. Therefore, if you want the message to be consumed by only one client then all of your consumers should be connected to a JMS queue rather than a topic.
We're using ActiveMQ (5.14.5).
We have a single producer, and multiple consumers on the same queue.
From time to time we set JMSXGroupID to group several messages together to be consumed on a single consumer. This works as expected.
In parallel, the producer continues to send non-grouped messages (i.e. without JMSXGroupID)
The problem:
We noticed that once a consumer was selected to process a specific group, it no longer gets the non-grouped messages. Even if it is completely idle. The non-grouped messages are always sent to the other consumers.
The rogue consumer returns to consume non-grouped messages only after we close the group that was assigned to it (by setting JMSXGroupSeq=-1).
Is this a normal behavior? We expected that non-grouped messages will continue to be delivered in the same round-robin fashion as usual, to all consumers.
We were unable to find a clear reference to this in ActiveMQ documentation.
There's a bit of a no-win situation for the message broker here. If there are active message groups in play, the the broker has to assume that further messages will be produced that fall into those groups. So a message consumer that has become bound to a particular group needs to remain available to consumer later messages of that group, rather than ungrouped messages. After all, an ungrouped message can be handled elsewhere, while a grouped message can't.
However, we also want to have a fair-ish distribution of messages between consumers. So it makes sense that a consumer that is bound to a group, or groups, could take some work when it is idle.
But how do we know it is idle? What happens if a consumer takes a bunch of ungrouped messages (and don't forget the default pre-fetch behaviour), and then new messages arrive that match its specific group?
The fact that closing a group restores the "group consumer" to default behaviour suggests to me that this is not a bug, but a deliberate attempt to make a reasonable compromise in a tricky situation. It seems reasonable to me to ask for a feature to be added, where "group consumers" can take part in ungrouped workload, but I would be inclined to see that as an enhancement.
Just my $0.02, of course.
I am begining to implement an ActiveMQ based messaging service to send worker tasks to various servers, however I am noticing that in the default mode, if no one is "listening" to a producer's topic, any message from that producer will be lost.
I.e.,
If Producer Senders Message with a live broker
But No Consumer is there to listen
Message goes no where
I would like instead for the Broker to hold on to messages until at least one listener receives it.
I am trying a couple ways of implementing this, but not sure on the most optimal/right way way:
Implement a Message Acknowledgement feature
(Caveat to this is I need the producer to wait on its listener after every message which seems very, very clunky and last resort...)
Implement the Session Transaction
(I am having trouble with this one, it sounds like the right thing to use here because of the word transaction, but I think it has more to do with the producer-broker interaction, not the producer-consumer)
Ideally, there is a mode to send a (or a set of) messages, and after sending a Boolean is returned stating if the message(s) were listened by at least one consumer.
Transactions and acknowlegdement conflict somehow with the general idea of a JMS topic.
Just use a queue instead of a topic. Access this queue using CLIENT_ACKNOWLEDGE or a transacted session. A worker task is to be processed by one worker only anyway, so the queue solves another problem.
If there was a special reason to use topics, you could consider a message driven bean (MDB) on the same host like the JMS provider (you could achieve this by using JBoss with its integrated HornetQ for example), but this is still not really correct.
Another possibility is to have both a topic and a queue. The latter is only for guaranteed delivery of each message.
This isn't really a typical messaging pattern. Typically, you have one receiver and a durable queue or multiple receivers with durable subscriptions to a topic. in either situation, each receiver will always receive the message. i don't really understand a use case where "at least one" receiver should receive it.
and yes, transactions only deal with the interactions between client and broker, not between client and eventual receiver(s).
I am trying to understand the use case of using Queue.
My understanding:
Queue means one-to-one. The only use case(if not rare, very few) would be: Message is intended for only one consume.
But even in those cases, I may want to use Topic (just to be future safe). The only extra caution would be to make subscriptions durable. Or, in special situations, I would use bridging / dispatcher mechanism.
Given above, I would always (or in most cases) want to publish to a topic. Subscriber can be either durable topic(s) or dispatched queue(s).
Please let me know what I am missing here or I am missing the original intent?
The design requirements on when to use queues are simple if you think in terms of real-world examples:
Submit online order (exactly-once processing to avoid charging credit
card twice)
Private peer-to-peer chat (exactly one receiver for each message)
Parallel task distribution (distribute tasks amongst many workers in a networked system)
...and examples for when to use topics...
News broadcast to multiple subscribers; notification service, stock ticker, etc.
Email client (unique durable subscriber; you still get emails when you're disconnected)
You said...
But even in those cases, I may want to use Topic (just to be future
safe). The only extra case I would have to do is to make (each)
subscription durable. Or, I special situations, I would use bridging /
dispatcher mechanism.
You're over-engineering the design. It's true, you can achieve exactly-once processing using a topic and durable subscriber, but you'd be limited to a single durable subscriber; the moment you start another subscriber for that topic, you'll get duplicate processing for the same message, not to mention, a single durable subscriber is hardly a solution that scales; it would be a bottleneck in your system for sure. With a queue, you can deploy 1000 receivers on 100 nodes for the same queue, and you'd still get exactly-once processing for a single message.
You said...
Give above, I would always (or in most cases) want to publish to a
topic. Subscriber can be either durable topic(s) or dispatched
queue(s).
Using a dispatched queue with a topic subscriber is sort of redundant. You basically get asynchronous dispatching when using queues, so why not just use a queue?...no reason to put a topic in front of it.
Queues and Topics in JMS represent two different models - point to point and publish/subscribe. Topics will keep a message until all clients receive them, all subscribers handling them. Queues will wait for the first consumer to pull the message, and consider it read at that point.
You are probably missing that both queues and topics can have multiple subscribers. A queue will deliver the message to one of potentially many subscribers, while a topic will deliver the message to all subscribers.
If you in your case are sure that there is only one subscriber, then a queue subscriber and a durable topic subscriber will behave similarly. I would rather look at such a scenario as a "special case".