I am trying to find an answer on of how to notify an EMS Publisher in case of a Subscriber failure.
In a case of Publisher->EMS server->Subscriber, if a Subscriber fails, I need to inform Publisher to take a corrective action.I am not bothered about durabilty/PERSIETENCE, my significance is of time. In Trading systems, If I send an market order to a Subscriber who in turn sends it to an exchange, if it fails, I need to make my Publisher publish the messages on a different topic to another Sunscriber(another exchange).
Any ideas is appreciated.
The tibjmsadmin.jar library contains methods to detect when subscribers disconnect. Easier than writing code, you can:
if you have Hawk, use the
tibjmsadmin.hma to write a Hawk rule
in the event a subscriber
disconnects, or
listen on the monitor
topic
$sys.monitor.connection.disconnect -
the body of the message tells you
which subscriber disconnected.
However these "monitoring" approaches to failing over the publisher have a significant problem - in the time it takes for you to detect the subscriber failure and redirect the publisher, some messages may get through and get stuck in the defunct queue. You don't wat this happening to any $10M trades!
EMS knows when subscribers are connected or not and you should take advantage of this.
Use a "distributed queue" and there shouldn't be any need to code logic into your application to switch to a new subscriber when it fails. This happens without message loss and maintains the order of messages. It is also good architectural practice to keep load balancing and failover logic out of your code and in the administration setup of your JMS provider.
Basically you setup multiple subscribers to a queue (each exchange represented by a subscriber). The default action will be for EMS to load-balance messages across your subscribers in a round-robin fashion. But you can set the queue to "exclusive" so that messages go to only one subscriber at a time. Then if that active subscriber fails, the messages are forwarded to another subscriber.
See the EMS manual for more details on all these topics.
Not sure if you have access, you could try looking at the ReceiverCount or ConsumerCount in either QueueInfo or TopicInfo - I believe you need the tibjms.admin package. May be you can query this before you publish and then selectively publish? Not sure what the overhead is.
Because of the nature of JMS, AFAIK no transaction states (unless you use a XA transaction - with all of that overhead) or acknowledgements will propagate through the EMS broker. I.e.acks are always between publisher and broker and consumer and broker.
Failing the above, you could try a separate ack topic for which the roles are reversed, but then the failure case is a timeout - I'm not sure this is sensible.
If you don't really care which exchange the order goes to - why not make the topic/queue exclusive and make both consumers attempt to consume - the first one to succeed will process all of the messages - if it dies, then the second one (which could be periodically retrying - may successfully connect).. Alternatively allow both to process orders off the queue - remember a message will only be ever processed by a single consumer...
I really cannot see the advantage of a decoupled messaging bus in your order flow... makes no sense to me..
Related
I have setup a JMS queue that is fed by a single producer, and consumed by 8 different consumers.
I would like to configure my queue/broker so that one message being delivered to a consumer blocks the queue until the consumer is done processing the message. During the processing of this first message, the following messages may not be delivered to another consumer. It doesn't matter which consumer processes which message, and it is acceptable for the same consumer to consume many messages in a row as long as when it dies another consumer is able to pick up the rest of the unprocessed messages.
In order to do this, I have configured all of my consumers to use the CLIENT acknowledgement mode, and I have coded them so that message.acknowledge() is called only at the end of the message processing.
My understanding was that this should be sufficient to satisfy my requirements.
However I am apparently wrong, because it looks like my brojer (OpenMQ) is delivering the messages to consumers as fast as possible, without waiting for the consumer acknowledgement. As a result, I get multiple messages processed in parallel, one for each consumer.
I'm obviously doing something wrong, but I can't figure out what.
As a workaround, I figure I could create a durable subscription with a fixed client ID shared between all my consumers. It would probably work by only allowing one consumer to even connect to the broker, but I can't shake the feeling that this is a rather ugly workaround.
Does anyone have an idea of how I should configure my Broker and/or my Client to make this possible?
I am begining to implement an ActiveMQ based messaging service to send worker tasks to various servers, however I am noticing that in the default mode, if no one is "listening" to a producer's topic, any message from that producer will be lost.
I.e.,
If Producer Senders Message with a live broker
But No Consumer is there to listen
Message goes no where
I would like instead for the Broker to hold on to messages until at least one listener receives it.
I am trying a couple ways of implementing this, but not sure on the most optimal/right way way:
Implement a Message Acknowledgement feature
(Caveat to this is I need the producer to wait on its listener after every message which seems very, very clunky and last resort...)
Implement the Session Transaction
(I am having trouble with this one, it sounds like the right thing to use here because of the word transaction, but I think it has more to do with the producer-broker interaction, not the producer-consumer)
Ideally, there is a mode to send a (or a set of) messages, and after sending a Boolean is returned stating if the message(s) were listened by at least one consumer.
Transactions and acknowlegdement conflict somehow with the general idea of a JMS topic.
Just use a queue instead of a topic. Access this queue using CLIENT_ACKNOWLEDGE or a transacted session. A worker task is to be processed by one worker only anyway, so the queue solves another problem.
If there was a special reason to use topics, you could consider a message driven bean (MDB) on the same host like the JMS provider (you could achieve this by using JBoss with its integrated HornetQ for example), but this is still not really correct.
Another possibility is to have both a topic and a queue. The latter is only for guaranteed delivery of each message.
This isn't really a typical messaging pattern. Typically, you have one receiver and a durable queue or multiple receivers with durable subscriptions to a topic. in either situation, each receiver will always receive the message. i don't really understand a use case where "at least one" receiver should receive it.
and yes, transactions only deal with the interactions between client and broker, not between client and eventual receiver(s).
I am trying to understand the use case of using Queue.
My understanding:
Queue means one-to-one. The only use case(if not rare, very few) would be: Message is intended for only one consume.
But even in those cases, I may want to use Topic (just to be future safe). The only extra caution would be to make subscriptions durable. Or, in special situations, I would use bridging / dispatcher mechanism.
Given above, I would always (or in most cases) want to publish to a topic. Subscriber can be either durable topic(s) or dispatched queue(s).
Please let me know what I am missing here or I am missing the original intent?
The design requirements on when to use queues are simple if you think in terms of real-world examples:
Submit online order (exactly-once processing to avoid charging credit
card twice)
Private peer-to-peer chat (exactly one receiver for each message)
Parallel task distribution (distribute tasks amongst many workers in a networked system)
...and examples for when to use topics...
News broadcast to multiple subscribers; notification service, stock ticker, etc.
Email client (unique durable subscriber; you still get emails when you're disconnected)
You said...
But even in those cases, I may want to use Topic (just to be future
safe). The only extra case I would have to do is to make (each)
subscription durable. Or, I special situations, I would use bridging /
dispatcher mechanism.
You're over-engineering the design. It's true, you can achieve exactly-once processing using a topic and durable subscriber, but you'd be limited to a single durable subscriber; the moment you start another subscriber for that topic, you'll get duplicate processing for the same message, not to mention, a single durable subscriber is hardly a solution that scales; it would be a bottleneck in your system for sure. With a queue, you can deploy 1000 receivers on 100 nodes for the same queue, and you'd still get exactly-once processing for a single message.
You said...
Give above, I would always (or in most cases) want to publish to a
topic. Subscriber can be either durable topic(s) or dispatched
queue(s).
Using a dispatched queue with a topic subscriber is sort of redundant. You basically get asynchronous dispatching when using queues, so why not just use a queue?...no reason to put a topic in front of it.
Queues and Topics in JMS represent two different models - point to point and publish/subscribe. Topics will keep a message until all clients receive them, all subscribers handling them. Queues will wait for the first consumer to pull the message, and consider it read at that point.
You are probably missing that both queues and topics can have multiple subscribers. A queue will deliver the message to one of potentially many subscribers, while a topic will deliver the message to all subscribers.
If you in your case are sure that there is only one subscriber, then a queue subscriber and a durable topic subscriber will behave similarly. I would rather look at such a scenario as a "special case".
I'm trying to understand the best way to coalesce or chunk incoming messages in RabbitMQ (using Spring AMQP or the Java client directly).
In other words I would like to take say 100 incoming messages and combine them as 1 and resend it to another queue in a reliable (correctly ACKed way). I believe this is called the aggregator pattern in EIP.
I know Spring Integration provides an aggregator solution but the implementation looks like its not fail safe (that is it looks like it has to ack and consume messages to build the coalesced message thus if you shutdown it down while its doing this you will loose messages?).
I can't comment directly on the Spring Integration library, so I'll speak generally in terms of RabbitMQ.
If you're not 100% convinced by the Spring Integration implementation of the Aggregator and are going to try to implement it yourself then I would recommend avoiding using tx which uses transactions under the hood in RabbitMQ.
Transactions in RabbitMQ are slow and you will definitely suffer performance problems if you're building a high traffic/throughput system.
Rather I would suggest you take a look at Publisher Confirms which is an extension to AMQP implemented in RabbitMQ. Here is an introduction to it when it was new http://www.rabbitmq.com/blog/2011/02/10/introducing-publisher-confirms/.
You will need to tweak the prefetch setting to get the performance right, take a look at http://www.rabbitmq.com/blog/2012/05/11/some-queuing-theory-throughput-latency-and-bandwidth/ for some details.
All the above gives you some background to help solve your problem. The implementation is rather straightforward.
When creating your consumer you will need to ensure you set it so that ACK is required.
Dequeue n messages, as you dequeue you will need to make note of the DeliveryTag for each message (this is used to ACK the message)
Aggregate the messages into a new message
Publish the new message
ACK each dequeued message
One thing to note is that if your consumer dies after 3 and before 4 has completed then those messages that weren't ACK'd will be reprocessed when it comes back to life
If you set the <amqp-inbound-channel-adapter/> tx-size attribute to 100, the container will ack every 100 messages so this should prevent message loss.
However, you might want to make the send of the aggregated message (on the 100th receive) transactional so you can confirm the broker has the message before the ack for the inbound messages.
I am fairly new to Java EE and JMS and am looking at doing an implementation using JMS.
Think of the following scenario:
Scenario
A user hits a servlet. A message is then put into a JMS server/Queue from this servlet. A response is then sent back to the user saying "Message Queued".
Option 1
The consumer/MDB receives the message from the JMS queue and processes it. This is normal operation and pretty standard.
Option 2
There is no consumer(for what ever reason) or the receiver is processing messages too slow. So what I would like is for the message in the queue to timeout. Once timed out, and email should be sent etc (email is just as an example).
Reading the API spec/Java EE 6 tutorial I have found in the QueuSender class
void send(Message message, int deliveryMode, int priority, long timeToLive)
So by settings the timeToLive the message will be evicted from the queue. The problem is that the is no "interface/call back" to know that the message was evicted. It just disappears. Or am I mistaken?
Another approach I thought of was for a thread to monitor the queue and evict messages that are "expired" and pull them from the queue. But I don't think that is possible, is it?
Any light shed on this matter would greatly be appreciated.
You have to make use of some implementation specific functionality to fulfill your requirements. The JMS specification does neither define which action is taken with a timed out message, nor does it offer you any reasonable criteria selection when polling messages from a queue.
Most (if not all) JMS implementations do however offer the concept of DLQs (dead letter queues). If a message cannot be delivered to a regular consumer or times out, the JMS implementation will most likely be able to move the message to a DLQ, which is basically also a regular queue with its own listener.
So, if you set up two queues, Q1 and Q2 and configure Q2 as a DLQ for Q1, you would do your normal request processing in a listener on Q1 and implement an additional listener for Q2 to do the error/timeout handling.
Synchronous interaction over JMS might be of help to you either. Basicly on the client side you:
send a message with a correlation id and time-to-live
receive a message (usually in the same thread) using the same correlation id and specifying timeout (time-to-live == timeout so if you treat it dead, it's really dead)
On the other side, server:
on an incoming message must fetch the correlation id
specify that correlation id for a response while sending it back to the client.
Of course server must be quick enough to fit the timeout/time-to-live threshold.
So on the client side you are always sure what's happend to the message that was sent.