Develop Reference

ActiveMQ browser needs long time for last .hasMoreElements()

I try to implement a queue browser for ActiveMQ.
The code shown below should display the text messages in the queue named 'Q1'. There are two messages in there. In general it works but the last e.hasMoreElements() call needs up to 20 seconds. I wanted to update the list every 500 millis. Why is that so slow?When i press 'update' in the browser view for http://localhost:8161/admin/browse.jsp?JMSDestination=Q1 e.hasMoreElements() returns immediately. What's going on here? How to achieve a 'realtime' view?
//init:
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("tcp://localhost:61616");
Connection connection = connectionFactory.createConnection();
connection.start();
Session session = connection.createSession(true, Session.CLIENT_ACKNOWLEDGE);
Queue queue = session.createQueue("Q1");
boolean run = true;
while (run) {
LOG.info("--------------------------------------------------------------------------------");
QueueBrowser browser = session.createBrowser(queue);
Enumeration e = browser.getEnumeration();
while (e.hasMoreElements()) { //<- very slow before returning false after last message. why?
Object o = e.nextElement();
if (o instanceof ActiveMQTextMessage) {
LOG.info(((ActiveMQTextMessage) o).getText());
} else {
LOG.info(o.toString());
}
}
Thread.sleep(500);
browser.close();
}
session.close();
connection.close();

After my previous comment, I've discovered that calling setTransactedIndividualAck(true) on the connection factory solves the problem
ActiveMQConnectionFactory cf2 = new ActiveMQConnectionFactory(...);
cf2.setTransactedIndividualAck(true);
I'm not sure that this is the right thing to do for the problem but at least it works now. See the message on the ActiveMQ user forum here:
http://activemq.2283324.n4.nabble.com/JMS-browser-getEnumeration-hasMoreElements-takes-15s-on-last-element-td4709969.html

I had the same issue. But changing the acknowledgment on the session eliminated the delay.
Try this:
Session session = connection.createSession(true, Session.AUTO_ACKNOWLEDGE);

I found that calling Session.commit() within my hasMoreElements() loop stopped the hanging using activemq-broker version 5.14.5:
while(enumeration.hasMoreElements()) {
final Message message = (Message)enumeration.nextElement();
final TextMessage textMessage = (TextMessage)message;
session.commit();
}
I did more research to see if this was a bug with ActiveMQ or not and found that activemq-broker version 5.15.1 did not hang, even without calling commit() after each iteration. All prior versions of the broker hanged on the final call to hasMoreElements(). It doesn't seem like the contributors deliberately fixed this particular issue, since the bug report on JIRA that the change referenced was for something different. The change that fixed this issue changed part of the iterate() method of the org.apache.activemq.broker.region.Queue class from:
// are we done browsing? no new messages paged
if (!added || browser.atMax()) {
browser.decrementQueueRef();
browserDispatches.remove(browserDispatch);
}
to
// are we done browsing? no new messages paged
if (!added || browser.atMax()) {
browser.decrementQueueRef();
browserDispatches.remove(browserDispatch);
} else {
wakeup();
}
To confirm this was the change that fixed the issue, I went to the previous version, 5.15.0 and forced wakeup() to be called using a debugger and the call to hasMoreElements() did not hang.

After some research and try i switched to the more up-to-date JMX technology. There are no performance issues while walking through the queues.
Some code:
JMXServiceURL url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi");
JMXConnector connector = JMXConnectorFactory.connect(url, null);
connector.connect();
connection = connector.getMBeanServerConnection();
ObjectName name = new ObjectName(getObjectNameByBrokerName(brokerName));
brokerMBean = (BrokerViewMBean) MBeanServerInvocationHandler.newProxyInstance(connection, name, BrokerViewMBean.class, true);
ObjectName[] objNames = brokerMBean.getQueues();
for (ObjectName objName : objNames) {
QueueViewMBean queueMBean = (QueueViewMBean) MBeanServerInvocationHandler.newProxyInstance(connection, objName, QueueViewMBean.class, true);
System.out.println(queueMBean.getName());
}
You have to activate jmx in the configuration. It's de-activated by default.

RabbitMQ: publish in node.js subscribe in Java

Given the following publisher in node.js and the following subscriber in java (this setup is fully functional) I have the following two questions:
What should I use as the third argument in queueBind and why? Why does it works as is ("test" is a random pick)?
Is there a way to specify queue in addition to exchange in rabbit.js? If yes then how? If not then why and which module should I use instead (code example would be welcome)?
// node.js
var context = require("rabbit.js").createContext();
var pub = context.socket('PUB');
pub.connect(config.exchange);
server.post("/message/:msg", function(req, res) {
pub.write(req.params.msg, 'utf8');
res.end();
});
// java
ConnectionFactory factory = new ConnectionFactory();
factory.setHost(host);
try {
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.exchangeDeclare(exchange, "fanout");
String queueName = channel.queueDeclare().getQueue();
channel.queueBind(queueName, exchange, "test"); // Question1: what should I use as the third argument and why?
// Question2: is there a way to configure rabbit.js with a queue name instead?
//channel.queueDeclare(queueName, false, false, false, null);
QueueingConsumer consumer = new QueueingConsumer(channel);
channel.basicConsume(queueName, true, consumer);
try {
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String message = new String(delivery.getBody());
LOG.info("Received message: " + message);
}
} catch (InterruptedException e) {
LOG.catching(e);
} finally {
channel.close();
connection.close();
}
} catch (IOException e) {
LOG.catching(e);
}

Own answer, what I've digged up so far:
The third argument, the routing key, is what is known as topic in rabbit.js. By supplying test I am only subscribing to messages send to the test topic or without a topic set (default in rabbit.js). If I were to use topic in the publisher as well, I could use pub.publish(topic, message, encoding) instead of pub.write(message, encoding) or supply it to the connect method
Does not look so and still do not know why really. The argument goes that rabbit.js is a higher-level library and it, therefore, makes certain simplifications. Why exactly this simplification is made I do not know. However, I primarily wanted to use a single exchange for multiple communication threads, which I can also achieve by using topics/routing keys. So not a big deal.

Delete top message from MQQueue

I am constructing a messaging system using MQSeries. For some reason, when I perform q.get(...), I am getting an exception thrown (I don't know the specific MQException). Below is the code causing the error:
private static MQGetMessageOptions GMO = new MQGetMessageOptions();
private static int GMO_OPTIONS = MQC.MQGMO_SYNCPOINT | MQC.MQGMO_WAIT;
GMO.options = GMO.options | GMO_OPTIONS;
GMO.waitInterval = MQC.MQWI_UNLIMITED;
MQEnvironment.hostname = args[0];
MQEnvironment.channel = args[2];
MQEnvironment.port = Integer.parseInt(args[1]);
MQQueueManager queueManager = new MQQueueManager(args[3])
MQMessage msg = new MQMessage();
MQQueue q = queueManager.accessQueue("qName1",MQC.MQOO_OUTPUT);
q.get(msg, GMO);
My plan is, when this error occurs, skip the message and delete it. To perform the delete I will call the following function:
private void deleteMsg(MQQueueManager queueManager, String queueName) throws MQException {
MQGetMessageOptions tempGmo = new MQGetMessageOptions();
tempGmo.options |= MQC.MQGMO_WAIT;
tempGmo.waitInterval = 1000;
MQQueue remover = queueManager.accessQueue(queueName, MQC.MQOO_INPUT_AS_Q_DEF);
remover.get(new MQMessage(), tempGmo);
queueManager.commit();
}
Would the remover.get() in my deleteMsg function also, in this specific scenario, fail for the same reason? Or does the option used to construct the MQQueue(MQC.MQOO_INPUT_AS_Q_DEF vs MQC.MQOO_OUTPUT) prevent it from also failing? If I am having trouble accessing my queue's message, how do I discard the top message and move to the next?
To shorten my question:
If I am unable to perform a get() on a given queue to retrieve a message, how can we delete that corrupt message on the same queue?
Thank you!

OMG!
MQQueue q = queueManager.accessQueue("qName1",MQC.MQOO_OUTPUT);
q.get(msg, GMO);
Your are opening a queue for output (writing) but you are trying to get a message. You have your shoes on the wrong feet!! Secondly, why aren't you catching the MQException that MQ would be throwing?? The exception would have included the reason code which would have given you the exact explanation to your issue.
Here's how you should be opening the queue for reading:
try
{
int oo = MQC.MQOO_INPUT_SHARED + MQC.MQOO_FAIL_IF_QUIESCING;
MQQueue q = queueManager.accessQueue("qName1",oo);
MQGetMessageOptions gmo = new MQGetMessageOptions();
gmo.options = MQC.MQGMO_NO_WAIT + MQC.MQGMO_FAIL_IF_QUIESCING;
q.get(msg, gmo);
}
catch (MQException e)
{
System.err.println(e.getLocalizedMessage() );
System.err.println("CC = " + e.completionCode + " - RC = " + e.reasonCode);
}
Also, make sure you use the appropriate "Fail if quiescing" option for the particular MQ API call.
Finally, look up "backout queue". If your application is having an issue with a message then the message should be moved to a backout queue and not simply deleted.

I do not why what you are doing does not work for you but I wonder why are you using proprietary API of MQ Series instead of using JMS API. In JMS terms remove top message just means receive the message, so call of session.receieve() does the work.
Using common JMS API has a lot of advantages. The main of them is that you can easily move from MQ Series to any other messaging solution without changing even one line of your code.

I wonder if the program compiled because there is no option called GMO_OPTIONS. All MQ constants are prefixed MQC

java websphere MQ

My aim is to put n number of messages in a for loop to a WebSphere MQ queue using WebSphere MQ java programming.
My java program will run as a standalone program.
If any exception in between , I need to rollback all the messages.
If no exception then I should commit all the messages .
The outside world should not see my messages in the queue until I complete fully.
How do I achieve this?
Updated with sample code as per reply from T.Rob:
Please check if sample code is fine ?
Does setting MQGMO_SYNCPOINT is only related to my program's invocation ?
(because similar programs running parallely will also be putting messages on the same queue and those messages should not gett affected by my program's SYNCPOINT.)
public void sendMsg() {
MQQueue queue = null;
MQQueueManager queueManager = null;
MQMessage mqMessage = null;
MQPutMessageOptions pmo = null;
System.out.println("Entering..");
try {
MQEnvironment.hostname = "x.x.x.x";
MQEnvironment.channel = "xxx.SVRCONN";
MQEnvironment.port = 9999;
queueManager = new MQQueueManager("XXXQMANAGER");
int openOptions = MQConstants.MQOO_OUTPUT;
queue = queueManager.accessQueue("XXX_QUEUENAME", openOptions, null, null, null);
pmo = new MQPutMessageOptions();
pmo.options = CMQC.MQGMO_SYNCPOINT;
String input = "testing";
System.out.println("sending messages....");
for (int i = 0; i < 10; i++) {
input = input + ": " + i;
mqMessage = new MQMessage();
mqMessage.writeString(input);
System.out.println("Putting message: " + i);
queue.put(mqMessage, pmo);
}
queueManager.commit();
System.out.println("Exiting..");
} catch (Exception e) {
e.printStackTrace();
try {
System.out.println("rolling back messages");
if (queueManager != null)
queueManager.backout();
} catch (MQException e1) {
e1.printStackTrace();
}
} finally {
try {
if (queue != null)
queue.close();
if (queueManager != null)
queueManager.close();
} catch (MQException e) {
e.printStackTrace();
}
}
}

WMQ supports both local and global (XA) units of work. The local units of work are available simply by specifying the option. Global XA transactions require a transaction manager, as mentioned by keithkreissl in another answer.
For what you described, a POJO doing messaging under syncpoint, specify MQC.MQGMO_SYNCPOINT in your MQGetMessageOptions. When you are ready to commit, issue the MQQManager.commit() or MQQManager.backout() call.
Note that the response and doc provided by ggrandes refers to the JMS and not Java classes. The Java classes use Java equivalents of the WMQ procedural API, can support many threads (doc) and even provide connection pooling (doc). Please refer to the Java documentation rather than the JMS documentation for the correct behavior. Also, I've linked to the WMQ V7.5 documentation which goes with the latest WMQ Java V7.5 client. The later clients have a lot more local functionality (tracing, flexible install path, MQClient.ini, etc.) and work with back-level QMgrs. It is highly recommended to be using the latest client and the download is free.

you only need to create a session with transaction enabled.
Session session;
// ...
boolean transacted = true;
session = connection.createSession(transacted, Session.AUTO_ACKNOWLEDGE);
try {
// ...do things...
session.commit();
} catch (Exception e) {
session.rollback();
}
// ...
WARN-NOTE: Sessions are not thread-safe ;-)
Doc Websphere MQ/JMS

If you have access to a transaction manager and more importantly an XATransaction wired up to your MQ access, you can start a transaction at the beginning of your message processing put all the messages on the queue then commit the transaction. Using the XATransactions it will not put any messages until the transaction commits. If you don't have access to that, you can do a little more plumbing by placing your messages in a local data object, wrap your code in a try/catch if no exceptions iterate through the local data object sending the messages. The issue with the later approach is that it will commit all your other processing but if a problem occurs in the sending of messages your other processing will not be rolled back.

Javamail performance

I've been using javamail to retrieve mails from IMAP server (currently GMail). Javamail retrieves list of messages (only ids) in a particular folder from server very fast, but when I actually fetch message (only envelop not even contents) it takes around 1 to 2 seconds for each message. What are the techniques should be used for fast retrieval?
here is my code:
try {
IMAPStore store = null;
if(store!=null&&store.isConnected())return;
Properties props = System.getProperties();
Session sessionIMAP = Session.getInstance(props, null);
try {
store = (IMAPStore) sessionIMAP.getStore("imaps");
store.connect("imap.gmail.com",993,"username#gmail.com","password");
} catch (Exception e) {
e.printStackTrace();
}
IMAPFolder folder = (IMAPFolder) store.getFolder("INBOX");
folder.open(Folder.READ_ONLY);
System.out.println("start");
Message[] msgs = folder.getMessages(1,10);
long ftime = System.currentTimeMillis();
FetchProfile fp=new FetchProfile();
fp.add(FetchProfile.Item.ENVELOPE);
folder.fetch(msgs, fp);
long time = System.currentTimeMillis();
System.out.println("fetch: "+(time-ftime));
for (Message message : msgs) {
System.out.println(message.getSubject());
Address[] from = message.getFrom();
for (Address address : from) {
System.out.println(address);
}
Address[] recipients = message.getAllRecipients();
for (Address address : recipients) {
System.out.println(address);
}
}
long newTime = System.currentTimeMillis();
System.out.println("convert: "+(newTime-time));
}catch (Exception e) {
e.printStackTrace();
}
}

I believe that Gmail throttles the IMAP message reads to one every second or so. You might be able to speed it up with multiple IMAP connections.

Please set the Property mail.imap.fetchsize with the required size. the default is 16k.
In case you increase the size of this property, retrieve speed will go up.
props.put("mail.imap.fetchsize", "3000000");
Note that if you're using the "imaps" protocol to access IMAP over SSL, all the properties would be named "mail.imaps.*".
Good Luck.
Yaniv

I'm not sure if this is a Javamail issue as much as it may be a Gmail issue. I have an application that retrieves mail from a number of sources, including Gmail, and Gmail is definitely the slowest. The Javamail api is pretty straightforward, but it would be hard to make suggestions without seeing what you are currently doing.

I'm running into the same thing. After profiling, I noticed that getBody was being called every time I tried to do a message.getFrom() like you are, even though I was only accessing fields that should be covered by the Envelope flag. See https://java.net/projects/javamail/forums/forum/topics/107956-gimap-efficiency-when-only-reading-headers