I have a requirement to consume the messages from the ActiveMQ topic and persist them in mongo. I am wondering if there is a way/configuration for consuming the messages in batch from the topic instead of reading messages one by one and making a DB call for every message.
I am imagining the end solution will do something like:
Consumes message in a batch size of 100
Use mongo bulk insert for saving the batch into DB
Send ACK to broker for successfully inserted messages and NAK for the failed message.
The JMS API only allows you to receive one message at a time whether that's via an asynchronous javax.jms.MessageListener or a synchronous call to javax.jms.MessageConsumer#receive() in JMS 1.1 or javax.jms.JMSConsumer.receive() in JMS 2. However, you can batch the receipt of multiple messages up using a transacted session. Here's what the javax.jms.Session JavaDoc says about transacted sessions:
A session may be specified as transacted. Each transacted session supports a single series of transactions. Each transaction groups a set of message sends and a set of message receives into an atomic unit of work. In effect, transactions organize a session's input message stream and output message stream into series of atomic units. When a transaction commits, its atomic unit of input is acknowledged and its associated atomic unit of output is sent. If a transaction rollback is done, the transaction's sent messages are destroyed and the session's input is automatically recovered.
So you can receive 100 messages individually using a transacted session, insert that data into Mongo, commit the transacted session or if there's a failure you can rollback the transacted session (which essentially acts as a negative acknowledgement). For example:
final int TX_SIZE = 100;
ConnectionFactory cf = new ActiveMQConnectionFactory("tcp://localhost:61616");
Connection connection = cf.createConnection();
Session session = connection.createSession(true, Session.SESSION_TRANSACTED);
Topic topic = session.createTopic("myTopic");
MessageConsumer consumer = session.createConsumer(topic);
connection.start();
while (true) {
List messages = new ArrayList<Message>();
for (int i = 0; i < TX_SIZE; i++) {
Message message = consumer.receive(1000);
if (message != null) {
messages.add(message);
} else {
break; // no more messages available for this batch
}
}
if (messages.size() > 0) {
try {
// bulk insert data from messages List into Mongo
session.commit();
} catch (Exception e) {
e.printStackTrace();
session.rollback();
}
} else {
break; // no more messages in the subscription
}
}
It's worth noting that if you are only using JMS transacted sessions and not full XA transactions there's going to be at least some risk of duplicates in Mongo (e.g. if your application crashes after successfully inserting data into Mongo but before committing the transacted session). XA transactions would mitigate this risk for you at the cost of a fair amount of additional complexity depending on your environment.
Lastly, if you run into performance limitations with ActiveMQ "Classic" consider using ActiveMQ Artemis, the next-generation message broker from ActiveMQ.
#Nabeel Ahmad you may be interested in checking out Virtual Topics in ActiveMQ. They provide an ability to use topics on the producer side, then use queues to consume. They are super helpful when wanting to scale consumption, as you have more features and observability using queues than topics on the consumer side.
Add this config to activemq.xml
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<virtualTopic name="VT.>" prefix="VQ.*." selectorAware="false"/>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
Then have producers send to: topic://VT.DATA
Then have consumer receive from: queue://VQ.CLIENT1.VT.DATA
As #Justin Bertram mentioned, batching reads can be done using a transacted session and committing every 100 or so messages.
Session session = connection.createSession(true, Session.SESSION_TRANSACTED);
MessageConsumer messageConsumer = session.createConsumer(session.createQueue("VQ.CLIENT1.VT.DATA");
Message message = null;
long count = 0l;
do {
message = messageConsumer.receive(2000l);
if(message != null) {
// check the message and publisher.send() to .DLQ if it is bad
// if message is good, send to Mongo
if(count % 100 == 0) {
// commit every 100 messages on the JMS-side
session.commit();
}
}
} while(message != null);
Related
I have setup a standalone HornetQ instance which is running locally. For testing purposes I have created a consumer using the HornetQ core API which will receive a message every 500 milliseconds.
I am facing a strange behaviour on the consumer side when my client connects and reads all the messages from queue and if I forced shutdown this (without properly closing the session/connection) then next time I start this consumer again it will read the old messages from the queue. Here is my consumer example:
// HornetQ Consumer Code
public void readMessage() {
ClientSession session = null;
try {
if (sf != null) {
session = sf.createSession(true, true);
ClientConsumer messageConsumer = session.createConsumer(JMS_QUEUE_NAME);
session.start();
while (true) {
ClientMessage messageReceived = messageConsumer.receive(1000);
if (messageReceived != null && messageReceived.getStringProperty(MESSAGE_PROPERTY_NAME) != null) {
System.out.println("Received JMS TextMessage:" + messageReceived.getStringProperty(MESSAGE_PROPERTY_NAME));
messageReceived.acknowledge();
}
Thread.sleep(500);
}
}
} catch (Exception e) {
LOGGER.error("Error while adding message by producer.", e);
} finally {
try {
session.close();
} catch (HornetQException e) {
LOGGER.error("Error while closing producer session,", e);
}
}
}
Can someone tell me why it is working like this, and what kind of configuration should I use in client/server side so that if a message read by consumer it will delete this from a queue?
You are not committing the session after the acknowledgements are complete, and you are not creating the session with auto-commit for acknowledgements enabled. Therefore, you should do one of the following:
Either explicitly call session.commit() after one or more invocations of acknowledge()
Or enable implicit auto-commit for acknowledgements by creating the session using sf.createSession(true,true) or sf.createSession(false,true) (the boolean which controls auto-commit for acknowledgements is the second one).
Keep in mind that when you enable auto-commit for acknowledgements there is an internal buffer which needs to reach a particular size before the acknowledgements are flushed to the broker. Batching acknowledgements like this can drastically improve performance for certain high-volume use-cases. By default you need to acknowledge 1,048,576 bytes worth of messages in order to flush the buffer and send the acknowledgements to the broker. You can change the size of this buffer by invoking setAckBatchSize on your ServerLocator instance or by using a different createSession method (e.g. sf.createSession(true, true, myAckBatchSize)).
If the acknowledgement buffer isn't flushed and your client crashes then the corresponding messages will still be in the queue when the client comes back. If the buffer hasn't reached its threshold it will still be flushed anyway when the consumer is closed gracefully.
I am trying to create an application which keeps on checking the number of queues up and running in activemq.
And Any way to check whether queue's are working or not i.e. if corrupted and not able to process messages.
Kindly suggest how to do it.
Thanks in Advance.
You can try following code.
public static void main(String[] args) throws Exception
{
// get the initial context
InitialContext ctx = new InitialContext();
// lookup the queue object
Queue queue = (Queue) ctx.lookup("queue/queue0");
// lookup the queue connection factory
QueueConnectionFactory connFactory = (QueueConnectionFactory) ctx.
lookup("queue/connectionFactory");
// create a queue connection
QueueConnection queueConn = connFactory.createQueueConnection();
// create a queue session
QueueSession queueSession = queueConn.createQueueSession(false,
Session.AUTO_ACKNOWLEDGE);
// create a queue browser
QueueBrowser queueBrowser = queueSession.createBrowser(queue);
// start the connection
queueConn.start();
// browse the messages
Enumeration e = queueBrowser.getEnumeration();
int numMsgs = 0;
// count number of messages
while (e.hasMoreElements()) {
Message message = (Message) e.nextElement();
numMsgs++;
}
System.out.println(queue + " has " + numMsgs + " messages");
// close the queue connection
queueConn.close();
}
You can ask for stats using the statistics plugin on the broker and the plain JMS api. I.e. to count the number of messages on FOO.BAR, send an empty message to ActiveMQ.Statistics.Destination.TEST.FOO and specify the replyTo header. In the response message, which is of typ MapMessage, you can find the message counter.
Another way is to browse only the first message of the queue using a simple queue browser (similar to the way praveen_programmer suggests) and check the timestamp of that message. If it's older than some threshold, you might have a problem with that consumer. I.e. no messages has been processed in the last hour/minute/day.
Yet another way is to use JMX, or preferably the jolokia REST/HTTP management api.
Just query the destination using http and you get a queue depth back:
To query the queue "q" on localhost, use the following api (you need to supply the user/password for the web console):
http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=q
Take a look at Advisory messages. You need to enable them in your config , but you can get a lot of useful about your current activemq instances info through simple JMS messaging. http://activemq.apache.org/advisory-message.html I was using them to highlight slow producer and consumer scenarios.
I am receiving messages from aActive MQ queue.
Is there a way to receive a number of messages in one time? or is that have to be done with a loop?
Further more, if i want to take say 30 messages run a procedure, and only if that procedure works return a message.acknowledge(); for all of them.
I mean i dont want to erase those 30 from the queue if the procedure fails.
Thanks.
You'll have to do it in a loop. Usually, it's best to use message-driven beans for consuming messages, but it's not suitable for this case, because they take message by message and you cannot specify the exact number. Thus, use MessageConsumer and manual transactions:
#Resource
UserTransaction utx;
#Resource(mappedName="jms/yourConnectionFactory");
ConnectionFactory cf;
#Resource(mappedName="jms/yourQueue");
Queue queue;
..
Connection conn = null;
Session s = null;
MessageConsumer mc = null;
try {
utx.begin();
conn = cf.createConnection();
s = conn.createSession(true, Session.CLIENT_ACKNOWLEDGE); //TRANSACTIONAL SESSION!
mc = s.createConsumer(queue);
conn.start(); // START CONNECTION'S DELIVERY OF INCOMING MESSAGES
for(int i=0; i<30; i++)
{
Message msg = mc.receive();
//BUSINESS LOGIC
}
utx.commit();
} catch(Exception ex) {
..
} finally { //CLOSE CONNECTION, SESSION AND MESSAGE CONSUMER
}
I don't have any experience in ActiveMQ. But I think in case of queue listeners, basic logic should be the same independent to the queue implementation.
For your first question I don't know any way of retrieving multiple messages from a queue. I think best way would be to fetch it one by one inside a loop.
For your second question, message will not be discarded from the queue till the underlying transaction which read the message commits. So you could read whole bunch of messages in a single transaction and roll it back in case of an error. It shouldn't erase any existing messages from the queue.
May I ask why do you need 30 messages to run a procedure. Usually when we use a queue, each message should be able to process independently.
There might be a stupid simple answer to this, but I'm trying to use ActiveMQ to pass messages between producers and consumers. I will have many producers and many consumers, but I want each message to be delivered only once among the consumers. This would seem to mean that I cannot use Topics, since they would deliver messages to all consumers who are listening, and I want only one Consumer to receive each message.
My problem is that I am able to receive messages, but the messages are not dequeued. So if I restart my consumer process, all of the messages are reprocessed. This answer seems pertinent but does not seem to apply since I can't create durable queue consumers, only durable topic consumers (unless I'm missing something in the API docs).
My code is as follows.
TopicConnectionFactory factory = new ActiveMQConnectionFactory(props.getProperty("mq.url"));
Connection conn = factory.createConnection();
Session session = conn.createSession(true, Session.CLIENT_ACKNOWLEDGE);
Queue queue = session.createQueue(props.getProperty("mq.source_queue"));
conn.start();
MessageConsumer consumer = session.createConsumer(queue);
Then later on
Message msg = consumer.receive();
msg.acknowledge();
if (!(msg instanceof TextMessage)) continue;
String msgStr = ((TextMessage)msg).getText();
This is my current code. I have tried with Session.AUTO_ACKNOWLEDGE and without msg.acknowledge(). Any working permutation of this code seems to retrieve the messages, but when I restart my consumer, all of the messages get received again, even if they have been received prior to the restart.
You created the session as a transacted Session and therefore need to call, session.commit if you want to inform the broker that all messages are now consumed and don't need to be redelivered. If you don't set the first argument to createSession to true then the Ack mode is respected otherwise its ignored, one of the oddities of the JMS API I'm afraid. If you do this:
ConnectionFactory factory = new ActiveMQConnectionFactory(props.getProperty("mq.url"));
Connection conn = factory.createConnection();
Session session = conn.createSession(false, Session.CLIENT_ACKNOWLEDGE);
Queue queue = session.createQueue(props.getProperty("mq.source_queue"));
conn.start();
MessageConsumer consumer = session.createConsumer(queue);
Then this would work:
Message msg = consumer.receive();
msg.acknowledge();
Otherwise you need to do:
Message msg = consumer.receive();
session.commit();
But keep in mind that for a single message transactions don't really make sense to client ack with no transaction is a better option.
I have a client that receives messages from a Queue. I currently have a MessageListener that implements onMessage().
Once the message is received, it is processed further then saved to a Database on the onMessage() method; the client then acknowledges the message receipt.
As long as the database is up there is no problem. But if the DB is down, the client will not acknowledge.
To cater for this, I want the client to be sending scheduled requests to the queue for any unacknowledged messages at scheduled intervals.
As it is, the only way I have of doing this is to restart the client which is not Ideal. Is there a way to trigger the queue to resend an unacknowledged message without a restart?
What i have in onMessage():
//code to connect to queue
try {
if (DB is available){
//process message
//save required details to DB
msg.acknowledge();
}
else{
//schedule to request same message later from queue
}
} catch (Exception e) {}
I think the standard behavior is already doing what you want: if the message broker is using the same database and the database is not available, it will not accept the messages and thus the client will spool them until the message broker is ready again.
If they do not share the same database and the message broker is on, it will spool the message and retry if onMessage throws an exception.
The message broker will try to resend according to its configurable policy.
After some more research, I have stumbled on the session.recover() which I can use to trigger redelivery. I have seen there is the RedeliveryPolicy class which I can use to set message resend options. Now my code looks like:
ConnectionFactory factory = new ActiveMQConnectionFactory(url);
RedeliveryPolicy policy = new RedeliveryPolicy();
policy.setBackOffMultiplier((short) 2);
policy.setRedeliveryDelay(30000);
policy.setInitialRedeliveryDelay(60000);
policy.setUseExponentialBackOff(true);
((ActiveMQConnectionFactory)factory).setRedeliveryPolicy(policy);
final Session session = connection.createSession(false,
Session.CLIENT_ACKNOWLEDGE);
...
...
...
..
//inside onMessage()
try {
if (DB is available){
//process message
//save required details to DB
msg.acknowledge();
}
else{
session.recover();
}
} catch (Exception e) {}