I'm new to JMS and would appreciate your help.
Everything I read about JMS gives me the impression that the object that "holds a transaction" is a session object, meaning that when we "session.commit()", everything that has happened since last commit is settled.
My question regards, how to handle the patter where I want to "transact messages", not the session. Imagine that you want to receive a large number of trades in a very concurrent scenario, and upon receiving any individual message and processing you'd like to "message.commit()". How can we implement this scenario ?
Would I have to keep creating QueueSessions for every message that I wanted to dequeue ? Isn't there too much overhead ?
Thanks in advance
It is correct, a JMS Session object is responsible for managing transactions. The Session.Commit() call commits all messages received/sent in that session. Similarly the Session.Rollback() rolls back messages in a session.
For your requirement you can use a non-transacted session with CLIENT_ACKNOWLEDGE option. Like:
createSession(false, Session.CLIENT_ACKNOWLEDGE);
A CLIENT_ACKNOWLEDGE session let's you do a message.Acknowledge which tells the messaging provider to remove that message from queue/topic.
But you have to keep in mind that the implementation of message.Acknowledge is implementation dependent. Some messaging providers allow per message acknowledge where as others do a acknowledge all messages. In per message acknowledge, only the message on which acknowledge is called, will be removed from messaging provider. All other messages received and not acknowledged will not be removed. Where as in the 'acknowledge all' type of implementation, calling acknowledge on one message will acknowledge all messages received before. This is same as doing a session.commit in a transacted session.
Related
Our project is to integrate two applications, using rest api of each, using JMS(to provide asynchronous nature) and spring batch to read bulk data from the JMS queue and process it and then post it to the receiving application.
I am a newbie to both JMS and spring batch. I have few basic questions to ask:
Which JMS model to ahead with-(PTP or Pub/Sub)
Can messages be read in bulk from the JMS queue(using JMSItemReader). If yes, can anyone pls provide with a code.
We want to acknowledge messages as 'read' once it is successfully posted (ie. read-process-write) to receiving application and not when it is read by the JMSItemReader. How can we achieve this?
The high level design diagram is below
PTP vs Pub/sub
The point to point method using a message queue is the most standard method to go. Especially in a batch application I can not see immediate reason to use Publish subscribe which presumes you have multiple consumers of the same messages.
Theoretically If multiple functions need to be executed over the same chunks of data you can organize the different processors as subscribers this way scaling the application, but this is pretty advanced usage scenario.
Can messages be read in bulk from JMS queue:
The JMS specification here only talks (vaguely might be misreading it) about bulk acknowledgment of messages, but it does not set a requirement over bulk delivery of messages.
CLIENT_ACKNOWLEDGE - With this option, a client acknowledges a message
by calling the message’s acknowledge method. Acknowledging a consumed
message automatically acknowledges the receipt of all messages that
have been delivered by its session.
Simply put the answer with respect of the bulk delivery is "If the JMS provider supports it, then yes, otherwise no"
Most providers allow bulk acknowledgment of messages
Here is the Oracles' interface doing that:
public interface com.sun.messaging.jms.Message {
void acknowledgeThisMessage() throws JMSException;
void acknowledgeUpThroughThisMessage() throws JMSException;
}
A combination of CLIENT_ACKNOWLEDGE . + invoking the method acknowledgeUpThroughThisMessage on a . message will acknowledge all messages received up to that moment in time.
Manual acknowledgment of messages:
This can be achieved through CLIENT_ACKNOWLEDGE and the acknowledge method on the Message. Here I will quote you again the javadoc of the acknowledge method which is also referencing one more time your second question and it talks about bulk acknowledgment of all messages up to a point.
void acknowledge()
throws JMSException Acknowledges all consumed messages of the session of this consumed message. All consumed JMS messages
support the acknowledge method for use when a client has specified
that its JMS session's consumed messages are to be explicitly
acknowledged. By invoking acknowledge on a consumed message, a client
acknowledges all messages consumed by the session that the message was
delivered to.
Calls to acknowledge are ignored for both transacted sessions and
sessions specified to use implicit acknowledgement modes.
A client may individually acknowledge each message as it is consumed,
or it may choose to acknowledge messages as an application-defined
group (which is done by calling acknowledge on the last received
message of the group, thereby acknowledging all messages consumed by
the session.)
Messages that have been received but not acknowledged may be
redelivered.
I am begining to implement an ActiveMQ based messaging service to send worker tasks to various servers, however I am noticing that in the default mode, if no one is "listening" to a producer's topic, any message from that producer will be lost.
I.e.,
If Producer Senders Message with a live broker
But No Consumer is there to listen
Message goes no where
I would like instead for the Broker to hold on to messages until at least one listener receives it.
I am trying a couple ways of implementing this, but not sure on the most optimal/right way way:
Implement a Message Acknowledgement feature
(Caveat to this is I need the producer to wait on its listener after every message which seems very, very clunky and last resort...)
Implement the Session Transaction
(I am having trouble with this one, it sounds like the right thing to use here because of the word transaction, but I think it has more to do with the producer-broker interaction, not the producer-consumer)
Ideally, there is a mode to send a (or a set of) messages, and after sending a Boolean is returned stating if the message(s) were listened by at least one consumer.
Transactions and acknowlegdement conflict somehow with the general idea of a JMS topic.
Just use a queue instead of a topic. Access this queue using CLIENT_ACKNOWLEDGE or a transacted session. A worker task is to be processed by one worker only anyway, so the queue solves another problem.
If there was a special reason to use topics, you could consider a message driven bean (MDB) on the same host like the JMS provider (you could achieve this by using JBoss with its integrated HornetQ for example), but this is still not really correct.
Another possibility is to have both a topic and a queue. The latter is only for guaranteed delivery of each message.
This isn't really a typical messaging pattern. Typically, you have one receiver and a durable queue or multiple receivers with durable subscriptions to a topic. in either situation, each receiver will always receive the message. i don't really understand a use case where "at least one" receiver should receive it.
and yes, transactions only deal with the interactions between client and broker, not between client and eventual receiver(s).
How to ensure message acknowledge deletes only messages upto the message on which acknowledge is called in a jms broker.
Currently I have a system which consumes from a jms queue and partially processes it.Sometime later a batch of these messages gets persisted by a different thread. I need to acknowledge on messages now. But the problem is I have to stop consuming the messages, otherwise acknowledging a previously received message will also acknowledge all other subsequent messages received.
In other words suppose I have 10 messages in a queue. I consume 7 of them, and then acknowledge on 5th message. This in turn removes all 7 messages received by consumer from the queue.Is there a way to only acknowledge and remove messages from queue till 5th message.
EDIT: I have tried creating two sessions and consuming from different sessions, but (with apache qpid atleast) this performs inconsistently. By inconsistently I mean, sometimes during the test it so happens that one consumer is able to receive messages, while the other doesn't receive at all, no matter how long you wait. This would have worked for me as a solution, but because of inconsistency can't use this as a solution.
I understand this post is old, but this answer should benefit those who stumble upon it later.
If you'd like fine grained control of which messages you'd like to acknowledge, the individual acknowledge method should help you. Using this acknowledgement mode you can ack individual messages in a session. Messages that have not been ack-ed will be redelivered.
This is not part of the spec, but most queue providers support it outside the spec.
Oracle
For more flexibility, Message Queue lets you customize the JMS
client-acknowledge mode. In client-acknowledge mode, the client
explicitly acknowledges message consumption by invoking the
acknowledge() method of a message object.
The standard behavior of
this method is to cause the session to acknowledge all messages that
have been consumed by any consumer in the session since the last time
the method was invoked. (That is, the session acknowledges the current
message and all previously unacknowledged messages, regardless of who
consumed them.)
In addition to the standard behavior specified by JMS, Message Queue
lets you use client-acknowledge mode to acknowledge one message at a
time.
public interface com.sun.messaging.jms.Message {
void acknowledgeThisMessage() throws JMSException;
void acknowledgeUpThroughThisMessage() throws JMSException;
}
ActiveMQ
One can imagine other acknowledge modes that would be useful too, for
example: CONSUMER_ACKNOWLEDGE where Message.acknowledge() would
acknowledge only messages received up on a particular MessageConsumer,
or CONSUMER_CHECKPOINT_ACKNOWLEDGE where Message.acknowledge() would
acknowledge only messages received up to and including the Message
instance on which the method was called.
But without embarking on all these various different possibilities,
would it be possible to consider just adding INDIVIDUAL_ACKNOWLEDGE
mode? This alone would make it possible for multithreaded applications
to achieve whatever behaviors they need.
connection.createQueueSession(false, ActiveMQSession.INDIVIDUAL_ACKNOWLEDGE);
I have not used QPID personally, however the documentation hints to the fact that individual message acks are possible.
Examples
# acknowledge all received messages
session.acknowledge
# acknowledge a single message
session.acknowledge :message => message
While processing a batch you can ack each message that is received and processed. If you encounter an exception, do not ack the message.
Acknowledging a message will make the queue manager to remove that message plus all other messages received before that message. It should not remove the messages which have not yet been received by an application. You may want to check your application on how you are acknowledging a message.
I am fairly new to Java EE and JMS and am looking at doing an implementation using JMS.
Think of the following scenario:
Scenario
A user hits a servlet. A message is then put into a JMS server/Queue from this servlet. A response is then sent back to the user saying "Message Queued".
Option 1
The consumer/MDB receives the message from the JMS queue and processes it. This is normal operation and pretty standard.
Option 2
There is no consumer(for what ever reason) or the receiver is processing messages too slow. So what I would like is for the message in the queue to timeout. Once timed out, and email should be sent etc (email is just as an example).
Reading the API spec/Java EE 6 tutorial I have found in the QueuSender class
void send(Message message, int deliveryMode, int priority, long timeToLive)
So by settings the timeToLive the message will be evicted from the queue. The problem is that the is no "interface/call back" to know that the message was evicted. It just disappears. Or am I mistaken?
Another approach I thought of was for a thread to monitor the queue and evict messages that are "expired" and pull them from the queue. But I don't think that is possible, is it?
Any light shed on this matter would greatly be appreciated.
You have to make use of some implementation specific functionality to fulfill your requirements. The JMS specification does neither define which action is taken with a timed out message, nor does it offer you any reasonable criteria selection when polling messages from a queue.
Most (if not all) JMS implementations do however offer the concept of DLQs (dead letter queues). If a message cannot be delivered to a regular consumer or times out, the JMS implementation will most likely be able to move the message to a DLQ, which is basically also a regular queue with its own listener.
So, if you set up two queues, Q1 and Q2 and configure Q2 as a DLQ for Q1, you would do your normal request processing in a listener on Q1 and implement an additional listener for Q2 to do the error/timeout handling.
Synchronous interaction over JMS might be of help to you either. Basicly on the client side you:
send a message with a correlation id and time-to-live
receive a message (usually in the same thread) using the same correlation id and specifying timeout (time-to-live == timeout so if you treat it dead, it's really dead)
On the other side, server:
on an incoming message must fetch the correlation id
specify that correlation id for a response while sending it back to the client.
Of course server must be quick enough to fit the timeout/time-to-live threshold.
So on the client side you are always sure what's happend to the message that was sent.
I am trying to find an answer on of how to notify an EMS Publisher in case of a Subscriber failure.
In a case of Publisher->EMS server->Subscriber, if a Subscriber fails, I need to inform Publisher to take a corrective action.I am not bothered about durabilty/PERSIETENCE, my significance is of time. In Trading systems, If I send an market order to a Subscriber who in turn sends it to an exchange, if it fails, I need to make my Publisher publish the messages on a different topic to another Sunscriber(another exchange).
Any ideas is appreciated.
The tibjmsadmin.jar library contains methods to detect when subscribers disconnect. Easier than writing code, you can:
if you have Hawk, use the
tibjmsadmin.hma to write a Hawk rule
in the event a subscriber
disconnects, or
listen on the monitor
topic
$sys.monitor.connection.disconnect -
the body of the message tells you
which subscriber disconnected.
However these "monitoring" approaches to failing over the publisher have a significant problem - in the time it takes for you to detect the subscriber failure and redirect the publisher, some messages may get through and get stuck in the defunct queue. You don't wat this happening to any $10M trades!
EMS knows when subscribers are connected or not and you should take advantage of this.
Use a "distributed queue" and there shouldn't be any need to code logic into your application to switch to a new subscriber when it fails. This happens without message loss and maintains the order of messages. It is also good architectural practice to keep load balancing and failover logic out of your code and in the administration setup of your JMS provider.
Basically you setup multiple subscribers to a queue (each exchange represented by a subscriber). The default action will be for EMS to load-balance messages across your subscribers in a round-robin fashion. But you can set the queue to "exclusive" so that messages go to only one subscriber at a time. Then if that active subscriber fails, the messages are forwarded to another subscriber.
See the EMS manual for more details on all these topics.
Not sure if you have access, you could try looking at the ReceiverCount or ConsumerCount in either QueueInfo or TopicInfo - I believe you need the tibjms.admin package. May be you can query this before you publish and then selectively publish? Not sure what the overhead is.
Because of the nature of JMS, AFAIK no transaction states (unless you use a XA transaction - with all of that overhead) or acknowledgements will propagate through the EMS broker. I.e.acks are always between publisher and broker and consumer and broker.
Failing the above, you could try a separate ack topic for which the roles are reversed, but then the failure case is a timeout - I'm not sure this is sensible.
If you don't really care which exchange the order goes to - why not make the topic/queue exclusive and make both consumers attempt to consume - the first one to succeed will process all of the messages - if it dies, then the second one (which could be periodically retrying - may successfully connect).. Alternatively allow both to process orders off the queue - remember a message will only be ever processed by a single consumer...
I really cannot see the advantage of a decoupled messaging bus in your order flow... makes no sense to me..