Develop Reference

ActiveMQ batch consumer

I have a requirement to consume the messages from the ActiveMQ topic and persist them in mongo. I am wondering if there is a way/configuration for consuming the messages in batch from the topic instead of reading messages one by one and making a DB call for every message.
I am imagining the end solution will do something like:
Consumes message in a batch size of 100
Use mongo bulk insert for saving the batch into DB
Send ACK to broker for successfully inserted messages and NAK for the failed message.

The JMS API only allows you to receive one message at a time whether that's via an asynchronous javax.jms.MessageListener or a synchronous call to javax.jms.MessageConsumer#receive() in JMS 1.1 or javax.jms.JMSConsumer.receive() in JMS 2. However, you can batch the receipt of multiple messages up using a transacted session. Here's what the javax.jms.Session JavaDoc says about transacted sessions:
A session may be specified as transacted. Each transacted session supports a single series of transactions. Each transaction groups a set of message sends and a set of message receives into an atomic unit of work. In effect, transactions organize a session's input message stream and output message stream into series of atomic units. When a transaction commits, its atomic unit of input is acknowledged and its associated atomic unit of output is sent. If a transaction rollback is done, the transaction's sent messages are destroyed and the session's input is automatically recovered.
So you can receive 100 messages individually using a transacted session, insert that data into Mongo, commit the transacted session or if there's a failure you can rollback the transacted session (which essentially acts as a negative acknowledgement). For example:
final int TX_SIZE = 100;
ConnectionFactory cf = new ActiveMQConnectionFactory("tcp://localhost:61616");
Connection connection = cf.createConnection();
Session session = connection.createSession(true, Session.SESSION_TRANSACTED);
Topic topic = session.createTopic("myTopic");
MessageConsumer consumer = session.createConsumer(topic);
connection.start();
while (true) {
List messages = new ArrayList<Message>();
for (int i = 0; i < TX_SIZE; i++) {
Message message = consumer.receive(1000);
if (message != null) {
messages.add(message);
} else {
break; // no more messages available for this batch
}
}
if (messages.size() > 0) {
try {
// bulk insert data from messages List into Mongo
session.commit();
} catch (Exception e) {
e.printStackTrace();
session.rollback();
}
} else {
break; // no more messages in the subscription
}
}
It's worth noting that if you are only using JMS transacted sessions and not full XA transactions there's going to be at least some risk of duplicates in Mongo (e.g. if your application crashes after successfully inserting data into Mongo but before committing the transacted session). XA transactions would mitigate this risk for you at the cost of a fair amount of additional complexity depending on your environment.
Lastly, if you run into performance limitations with ActiveMQ "Classic" consider using ActiveMQ Artemis, the next-generation message broker from ActiveMQ.

#Nabeel Ahmad you may be interested in checking out Virtual Topics in ActiveMQ. They provide an ability to use topics on the producer side, then use queues to consume. They are super helpful when wanting to scale consumption, as you have more features and observability using queues than topics on the consumer side.
Add this config to activemq.xml
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<virtualTopic name="VT.>" prefix="VQ.*." selectorAware="false"/>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
Then have producers send to: topic://VT.DATA
Then have consumer receive from: queue://VQ.CLIENT1.VT.DATA
As #Justin Bertram mentioned, batching reads can be done using a transacted session and committing every 100 or so messages.
Session session = connection.createSession(true, Session.SESSION_TRANSACTED);
MessageConsumer messageConsumer = session.createConsumer(session.createQueue("VQ.CLIENT1.VT.DATA");
Message message = null;
long count = 0l;
do {
message = messageConsumer.receive(2000l);
if(message != null) {
// check the message and publisher.send() to .DLQ if it is bad
// if message is good, send to Mongo
if(count % 100 == 0) {
// commit every 100 messages on the JMS-side
session.commit();
}
}
} while(message != null);

Acknowledged messages are redelivered when crashed core client reconnects

I have setup a standalone HornetQ instance which is running locally. For testing purposes I have created a consumer using the HornetQ core API which will receive a message every 500 milliseconds.
I am facing a strange behaviour on the consumer side when my client connects and reads all the messages from queue and if I forced shutdown this (without properly closing the session/connection) then next time I start this consumer again it will read the old messages from the queue. Here is my consumer example:
// HornetQ Consumer Code
public void readMessage() {
ClientSession session = null;
try {
if (sf != null) {
session = sf.createSession(true, true);
ClientConsumer messageConsumer = session.createConsumer(JMS_QUEUE_NAME);
session.start();
while (true) {
ClientMessage messageReceived = messageConsumer.receive(1000);
if (messageReceived != null && messageReceived.getStringProperty(MESSAGE_PROPERTY_NAME) != null) {
System.out.println("Received JMS TextMessage:" + messageReceived.getStringProperty(MESSAGE_PROPERTY_NAME));
messageReceived.acknowledge();
}
Thread.sleep(500);
}
}
} catch (Exception e) {
LOGGER.error("Error while adding message by producer.", e);
} finally {
try {
session.close();
} catch (HornetQException e) {
LOGGER.error("Error while closing producer session,", e);
}
}
}
Can someone tell me why it is working like this, and what kind of configuration should I use in client/server side so that if a message read by consumer it will delete this from a queue?

You are not committing the session after the acknowledgements are complete, and you are not creating the session with auto-commit for acknowledgements enabled. Therefore, you should do one of the following:
Either explicitly call session.commit() after one or more invocations of acknowledge()
Or enable implicit auto-commit for acknowledgements by creating the session using sf.createSession(true,true) or sf.createSession(false,true) (the boolean which controls auto-commit for acknowledgements is the second one).
Keep in mind that when you enable auto-commit for acknowledgements there is an internal buffer which needs to reach a particular size before the acknowledgements are flushed to the broker. Batching acknowledgements like this can drastically improve performance for certain high-volume use-cases. By default you need to acknowledge 1,048,576 bytes worth of messages in order to flush the buffer and send the acknowledgements to the broker. You can change the size of this buffer by invoking setAckBatchSize on your ServerLocator instance or by using a different createSession method (e.g. sf.createSession(true, true, myAckBatchSize)).
If the acknowledgement buffer isn't flushed and your client crashes then the corresponding messages will still be in the queue when the client comes back. If the buffer hasn't reached its threshold it will still be flushed anyway when the consumer is closed gracefully.

Receiving number of messages from a queue

I am receiving messages from aActive MQ queue.
Is there a way to receive a number of messages in one time? or is that have to be done with a loop?
Further more, if i want to take say 30 messages run a procedure, and only if that procedure works return a message.acknowledge(); for all of them.
I mean i dont want to erase those 30 from the queue if the procedure fails.
Thanks.

You'll have to do it in a loop. Usually, it's best to use message-driven beans for consuming messages, but it's not suitable for this case, because they take message by message and you cannot specify the exact number. Thus, use MessageConsumer and manual transactions:
#Resource
UserTransaction utx;
#Resource(mappedName="jms/yourConnectionFactory");
ConnectionFactory cf;
#Resource(mappedName="jms/yourQueue");
Queue queue;
..
Connection conn = null;
Session s = null;
MessageConsumer mc = null;
try {
utx.begin();
conn = cf.createConnection();
s = conn.createSession(true, Session.CLIENT_ACKNOWLEDGE); //TRANSACTIONAL SESSION!
mc = s.createConsumer(queue);
conn.start(); // START CONNECTION'S DELIVERY OF INCOMING MESSAGES
for(int i=0; i<30; i++)
{
Message msg = mc.receive();
//BUSINESS LOGIC
}
utx.commit();
} catch(Exception ex) {
..
} finally { //CLOSE CONNECTION, SESSION AND MESSAGE CONSUMER
}

I don't have any experience in ActiveMQ. But I think in case of queue listeners, basic logic should be the same independent to the queue implementation.
For your first question I don't know any way of retrieving multiple messages from a queue. I think best way would be to fetch it one by one inside a loop.
For your second question, message will not be discarded from the queue till the underlying transaction which read the message commits. So you could read whole bunch of messages in a single transaction and roll it back in case of an error. It shouldn't erase any existing messages from the queue.
May I ask why do you need 30 messages to run a procedure. Usually when we use a queue, each message should be able to process independently.

JMS Message Sending After JMS Server Restart During Runtime

I am having a problem about JMS. The problem is, I have an application and it is trying to send a message through JMS , but after JMS server restart, it throws exception as when the server was down time. It is not reconnecting.
It is completely fine without a restart of JMS server and I am using weblogic 10.x.
Is it a problem about JMS configuration?
Thanks

Can you post your code how you are sending messages?
If you attempt to send a message when the JMS server is down, you will probably have an exception, and have to deal with that. When you attempt to send a message the next time, when the JMS server is restarted and running, you probably create a new connection from your connection factory. Reconnection will happend then:
// Let's say, you inject CF and Dest.
#Resource(lookup = "jms/ConnectionFactory")
private static ConnectionFactory cf;
#Resource(lookup = "jms/MyQueue")
private static Destination dest;
public void sendMessage(){ // called every time you need to send a message.
try{
Connection con = cf.createConnection(); // will reconnect, otherwise pooled.
Session sess = con.createSession(false,Session.AUTO_ACKNOWLEDGE);
MessageProducer prod = sess.createProducer(dest);
Message msg = sess.createTextMessage("Hello, World");
prod.send(msg);
sess.close();
con.close();
}catch(Exception e){
// handle errors
}
}
However, there are some built in failover/reconnecting features in Weblogic JMS, take a look at this page:
Oracle code listing and documentation about reconnecting JMS producers

Unable to get ActiveMQ to dequeue

There might be a stupid simple answer to this, but I'm trying to use ActiveMQ to pass messages between producers and consumers. I will have many producers and many consumers, but I want each message to be delivered only once among the consumers. This would seem to mean that I cannot use Topics, since they would deliver messages to all consumers who are listening, and I want only one Consumer to receive each message.
My problem is that I am able to receive messages, but the messages are not dequeued. So if I restart my consumer process, all of the messages are reprocessed. This answer seems pertinent but does not seem to apply since I can't create durable queue consumers, only durable topic consumers (unless I'm missing something in the API docs).
My code is as follows.
TopicConnectionFactory factory = new ActiveMQConnectionFactory(props.getProperty("mq.url"));
Connection conn = factory.createConnection();
Session session = conn.createSession(true, Session.CLIENT_ACKNOWLEDGE);
Queue queue = session.createQueue(props.getProperty("mq.source_queue"));
conn.start();
MessageConsumer consumer = session.createConsumer(queue);
Then later on
Message msg = consumer.receive();
msg.acknowledge();
if (!(msg instanceof TextMessage)) continue;
String msgStr = ((TextMessage)msg).getText();
This is my current code. I have tried with Session.AUTO_ACKNOWLEDGE and without msg.acknowledge(). Any working permutation of this code seems to retrieve the messages, but when I restart my consumer, all of the messages get received again, even if they have been received prior to the restart.

You created the session as a transacted Session and therefore need to call, session.commit if you want to inform the broker that all messages are now consumed and don't need to be redelivered. If you don't set the first argument to createSession to true then the Ack mode is respected otherwise its ignored, one of the oddities of the JMS API I'm afraid. If you do this:
ConnectionFactory factory = new ActiveMQConnectionFactory(props.getProperty("mq.url"));
Connection conn = factory.createConnection();
Session session = conn.createSession(false, Session.CLIENT_ACKNOWLEDGE);
Queue queue = session.createQueue(props.getProperty("mq.source_queue"));
conn.start();
MessageConsumer consumer = session.createConsumer(queue);
Then this would work:
Message msg = consumer.receive();
msg.acknowledge();
Otherwise you need to do:
Message msg = consumer.receive();
session.commit();
But keep in mind that for a single message transactions don't really make sense to client ack with no transaction is a better option.