I am receiving messages from aActive MQ queue.
Is there a way to receive a number of messages in one time? or is that have to be done with a loop?
Further more, if i want to take say 30 messages run a procedure, and only if that procedure works return a message.acknowledge(); for all of them.
I mean i dont want to erase those 30 from the queue if the procedure fails.
Thanks.
You'll have to do it in a loop. Usually, it's best to use message-driven beans for consuming messages, but it's not suitable for this case, because they take message by message and you cannot specify the exact number. Thus, use MessageConsumer and manual transactions:
#Resource
UserTransaction utx;
#Resource(mappedName="jms/yourConnectionFactory");
ConnectionFactory cf;
#Resource(mappedName="jms/yourQueue");
Queue queue;
..
Connection conn = null;
Session s = null;
MessageConsumer mc = null;
try {
utx.begin();
conn = cf.createConnection();
s = conn.createSession(true, Session.CLIENT_ACKNOWLEDGE); //TRANSACTIONAL SESSION!
mc = s.createConsumer(queue);
conn.start(); // START CONNECTION'S DELIVERY OF INCOMING MESSAGES
for(int i=0; i<30; i++)
{
Message msg = mc.receive();
//BUSINESS LOGIC
}
utx.commit();
} catch(Exception ex) {
..
} finally { //CLOSE CONNECTION, SESSION AND MESSAGE CONSUMER
}
I don't have any experience in ActiveMQ. But I think in case of queue listeners, basic logic should be the same independent to the queue implementation.
For your first question I don't know any way of retrieving multiple messages from a queue. I think best way would be to fetch it one by one inside a loop.
For your second question, message will not be discarded from the queue till the underlying transaction which read the message commits. So you could read whole bunch of messages in a single transaction and roll it back in case of an error. It shouldn't erase any existing messages from the queue.
May I ask why do you need 30 messages to run a procedure. Usually when we use a queue, each message should be able to process independently.
Related
I have a requirement to consume the messages from the ActiveMQ topic and persist them in mongo. I am wondering if there is a way/configuration for consuming the messages in batch from the topic instead of reading messages one by one and making a DB call for every message.
I am imagining the end solution will do something like:
Consumes message in a batch size of 100
Use mongo bulk insert for saving the batch into DB
Send ACK to broker for successfully inserted messages and NAK for the failed message.
The JMS API only allows you to receive one message at a time whether that's via an asynchronous javax.jms.MessageListener or a synchronous call to javax.jms.MessageConsumer#receive() in JMS 1.1 or javax.jms.JMSConsumer.receive() in JMS 2. However, you can batch the receipt of multiple messages up using a transacted session. Here's what the javax.jms.Session JavaDoc says about transacted sessions:
A session may be specified as transacted. Each transacted session supports a single series of transactions. Each transaction groups a set of message sends and a set of message receives into an atomic unit of work. In effect, transactions organize a session's input message stream and output message stream into series of atomic units. When a transaction commits, its atomic unit of input is acknowledged and its associated atomic unit of output is sent. If a transaction rollback is done, the transaction's sent messages are destroyed and the session's input is automatically recovered.
So you can receive 100 messages individually using a transacted session, insert that data into Mongo, commit the transacted session or if there's a failure you can rollback the transacted session (which essentially acts as a negative acknowledgement). For example:
final int TX_SIZE = 100;
ConnectionFactory cf = new ActiveMQConnectionFactory("tcp://localhost:61616");
Connection connection = cf.createConnection();
Session session = connection.createSession(true, Session.SESSION_TRANSACTED);
Topic topic = session.createTopic("myTopic");
MessageConsumer consumer = session.createConsumer(topic);
connection.start();
while (true) {
List messages = new ArrayList<Message>();
for (int i = 0; i < TX_SIZE; i++) {
Message message = consumer.receive(1000);
if (message != null) {
messages.add(message);
} else {
break; // no more messages available for this batch
}
}
if (messages.size() > 0) {
try {
// bulk insert data from messages List into Mongo
session.commit();
} catch (Exception e) {
e.printStackTrace();
session.rollback();
}
} else {
break; // no more messages in the subscription
}
}
It's worth noting that if you are only using JMS transacted sessions and not full XA transactions there's going to be at least some risk of duplicates in Mongo (e.g. if your application crashes after successfully inserting data into Mongo but before committing the transacted session). XA transactions would mitigate this risk for you at the cost of a fair amount of additional complexity depending on your environment.
Lastly, if you run into performance limitations with ActiveMQ "Classic" consider using ActiveMQ Artemis, the next-generation message broker from ActiveMQ.
#Nabeel Ahmad you may be interested in checking out Virtual Topics in ActiveMQ. They provide an ability to use topics on the producer side, then use queues to consume. They are super helpful when wanting to scale consumption, as you have more features and observability using queues than topics on the consumer side.
Add this config to activemq.xml
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<virtualTopic name="VT.>" prefix="VQ.*." selectorAware="false"/>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
Then have producers send to: topic://VT.DATA
Then have consumer receive from: queue://VQ.CLIENT1.VT.DATA
As #Justin Bertram mentioned, batching reads can be done using a transacted session and committing every 100 or so messages.
Session session = connection.createSession(true, Session.SESSION_TRANSACTED);
MessageConsumer messageConsumer = session.createConsumer(session.createQueue("VQ.CLIENT1.VT.DATA");
Message message = null;
long count = 0l;
do {
message = messageConsumer.receive(2000l);
if(message != null) {
// check the message and publisher.send() to .DLQ if it is bad
// if message is good, send to Mongo
if(count % 100 == 0) {
// commit every 100 messages on the JMS-side
session.commit();
}
}
} while(message != null);
I am trying to create an application which keeps on checking the number of queues up and running in activemq.
And Any way to check whether queue's are working or not i.e. if corrupted and not able to process messages.
Kindly suggest how to do it.
Thanks in Advance.
You can try following code.
public static void main(String[] args) throws Exception
{
// get the initial context
InitialContext ctx = new InitialContext();
// lookup the queue object
Queue queue = (Queue) ctx.lookup("queue/queue0");
// lookup the queue connection factory
QueueConnectionFactory connFactory = (QueueConnectionFactory) ctx.
lookup("queue/connectionFactory");
// create a queue connection
QueueConnection queueConn = connFactory.createQueueConnection();
// create a queue session
QueueSession queueSession = queueConn.createQueueSession(false,
Session.AUTO_ACKNOWLEDGE);
// create a queue browser
QueueBrowser queueBrowser = queueSession.createBrowser(queue);
// start the connection
queueConn.start();
// browse the messages
Enumeration e = queueBrowser.getEnumeration();
int numMsgs = 0;
// count number of messages
while (e.hasMoreElements()) {
Message message = (Message) e.nextElement();
numMsgs++;
}
System.out.println(queue + " has " + numMsgs + " messages");
// close the queue connection
queueConn.close();
}
You can ask for stats using the statistics plugin on the broker and the plain JMS api. I.e. to count the number of messages on FOO.BAR, send an empty message to ActiveMQ.Statistics.Destination.TEST.FOO and specify the replyTo header. In the response message, which is of typ MapMessage, you can find the message counter.
Another way is to browse only the first message of the queue using a simple queue browser (similar to the way praveen_programmer suggests) and check the timestamp of that message. If it's older than some threshold, you might have a problem with that consumer. I.e. no messages has been processed in the last hour/minute/day.
Yet another way is to use JMX, or preferably the jolokia REST/HTTP management api.
Just query the destination using http and you get a queue depth back:
To query the queue "q" on localhost, use the following api (you need to supply the user/password for the web console):
http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=q
Take a look at Advisory messages. You need to enable them in your config , but you can get a lot of useful about your current activemq instances info through simple JMS messaging. http://activemq.apache.org/advisory-message.html I was using them to highlight slow producer and consumer scenarios.
There might be a stupid simple answer to this, but I'm trying to use ActiveMQ to pass messages between producers and consumers. I will have many producers and many consumers, but I want each message to be delivered only once among the consumers. This would seem to mean that I cannot use Topics, since they would deliver messages to all consumers who are listening, and I want only one Consumer to receive each message.
My problem is that I am able to receive messages, but the messages are not dequeued. So if I restart my consumer process, all of the messages are reprocessed. This answer seems pertinent but does not seem to apply since I can't create durable queue consumers, only durable topic consumers (unless I'm missing something in the API docs).
My code is as follows.
TopicConnectionFactory factory = new ActiveMQConnectionFactory(props.getProperty("mq.url"));
Connection conn = factory.createConnection();
Session session = conn.createSession(true, Session.CLIENT_ACKNOWLEDGE);
Queue queue = session.createQueue(props.getProperty("mq.source_queue"));
conn.start();
MessageConsumer consumer = session.createConsumer(queue);
Then later on
Message msg = consumer.receive();
msg.acknowledge();
if (!(msg instanceof TextMessage)) continue;
String msgStr = ((TextMessage)msg).getText();
This is my current code. I have tried with Session.AUTO_ACKNOWLEDGE and without msg.acknowledge(). Any working permutation of this code seems to retrieve the messages, but when I restart my consumer, all of the messages get received again, even if they have been received prior to the restart.
You created the session as a transacted Session and therefore need to call, session.commit if you want to inform the broker that all messages are now consumed and don't need to be redelivered. If you don't set the first argument to createSession to true then the Ack mode is respected otherwise its ignored, one of the oddities of the JMS API I'm afraid. If you do this:
ConnectionFactory factory = new ActiveMQConnectionFactory(props.getProperty("mq.url"));
Connection conn = factory.createConnection();
Session session = conn.createSession(false, Session.CLIENT_ACKNOWLEDGE);
Queue queue = session.createQueue(props.getProperty("mq.source_queue"));
conn.start();
MessageConsumer consumer = session.createConsumer(queue);
Then this would work:
Message msg = consumer.receive();
msg.acknowledge();
Otherwise you need to do:
Message msg = consumer.receive();
session.commit();
But keep in mind that for a single message transactions don't really make sense to client ack with no transaction is a better option.
I want to build a simple consumer program (in java) to get all messages stocked in an ActiveMQ subject.
I have a producer which send TextMessage in the queue.
But I don't know how to start to write my consumer for retrieving old messages and wait for new one.
If you have an example, thanks!
This is my Producer: http://pastebin.com/uRy9D8mY
This is my Consumer: http://pastebin.com/bZh4r66e
When I run my producer before my consumer, then run the consumer, I got nothing.
When I run my consumer then my producer, I add 72 messages in the queue but my consumer got only 24 message...
I suggest reading this tutorial (as does Apache ActiveMQ) SUN Jms tutorial
There are many ways to write JMS/ActiveMQ programs, using various frameworks such as Spring, or by using plain java.
Essentially, write a listener class like this:
public class MyListener implements MessageListener{
public void onMessage(Message message){
// Read and handle message here.
}
}
Since you already are producing message, I assume you have connection up and running.
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
consumer = session.createConsumer("MyQueue");
listener = new MyListener ();
consumer.setMessageListener(listener);
connection.start();
// At this point, messages should arrive from the queue to your listener.
Then there are some error handling code not included in this example, but you should be able to figure it out with the help of the tutorial and JMS documentation.
Using the code given below you can read all the messages en-queued in the queue.
In this code the while loop is an unending loop it will iterate all
the messages in the queue.
Once there is no messages in the queue it will wait for 5 seconds then automatically stops the connection and breaks
the loop.
If you required an unending consumer which will read all the messages whenever newly added to the queue, then remove the else part, so the program will not terminate.
ConnectionFactory factory = new ActiveMQConnectionFactory("tcp://localhost:61616");
Connection con = factory.createConnection();
Session session = con.createSession(false, Session.AUTO_ACKNOWLEDGE);
Queue queue = session.createQueue("tmp_queue2");
MessageConsumer consumer = session.createConsumer(queue);
con.start();
while (true) {
Message msg = consumer.receive(5000);
if (msg instanceof TextMessage) {
TextMessage tm = (TextMessage) msg;
System.out.println(tm.getText());
}
else{
System.out.println("Queue Empty");
con.stop();
break;
}
}
Hope this consumer program will helps people who were new to ActiveMQ.
I am trying to handle flow control situation on producer end.
I have a queue on a qpid-broker with a max queue-size set. Also have flow_stop_count and flow_resume_count set on the queue.
now at the producer keeps on continuously producing messages until this flow_stop_count is reached. Upon breach of this count, an exception is thrown which is handled by Exception listener.
Now sometime later the consumer on queue will catch up and the flow_resume_count will be reached. The question is how does the producer know of this event.
Here's a sample code of the producer
connection connection = connectionFactory.createConnection();
connection.setExceptionListenr(new MyExceptionListerner());
connection.start();
Session session = connection.createSession(false,Session.CLIENT_ACKNOWLEDGE);
Queue queue = (Queue)context.lookup("Test");
MessageProducer producer = session.createProducer(queue);
while(notStopped){
while(suspend){//---------------------------how to resume this flag???
Thread.sleep(1000);
}
TextMessage message = session.createTextMessage();
message.setText("TestMessage");
producer.send(message);
}
session.close();
connection.close();
and for the exception listener
private class MyExceptionListener implements ExceptionListener {
public void onException(JMSException e) {
System.out.println("got exception:" + e.getMessage());
suspend=true;
}
}
Now the exceptionlistener is a generic listener for exceptions, so it should not be a good idea to suspend the producer flow through that.
What I need is perhaps some method on the producer level , something like produer.isFlowStopped() which I can use to check before sending a message. Does such a functionality exist in qpid api.
There is some documentation on the qpid website which suggest this can be done. But I couldn't find any examples of this being done anywhere.
Is there some standard way of handling this kind of scenario.
From what I have read from the Apache QPid documentation it seems that the flow_resume_count and flow_stop_count will cause the producers to start getting blocked.
Therefore the only option would be to software wise to poll at regular intervals until the messages start flowing again.
Extract from here.
If a producer sends to a queue which is overfull, the broker will respond by instructing the client not to send any more messages. The impact of this is that any future attempts to send will block until the broker rescinds the flow control order.
While blocking the client will periodically log the fact that it is blocked waiting on flow control.
WARN AMQSession - Broker enforced flow control has been enforced
WARN AMQSession - Message send delayed by 5s due to broker enforced flow control
WARN AMQSession - Message send delayed by 10s due to broker enforced flow control
After a set period the send will timeout and throw a JMSException to the calling code.
ERROR AMQSession - Message send failed due to timeout waiting on broker enforced flow control.
From this documentation it implicates that the software managing the producer would then have to self manage. So basically when you receive an exception that the queue is overfull you will need to back off and most likely poll and reattempt to send your messages.
You can try setting the capacity (size in bytes at which the queue is thought to be full ) and flowResumeCapacity (the queue size at which producers are unflowed) properties for a queue.
send() will then be blocked if the size exceeds the capacity value.
You can have a look at this test case file in the repo to get an idea.
Producer flow control is not yet implemented on the JMS client.
See https://issues.apache.org/jira/browse/QPID-3388