I have a simple java program to transfer messages from queue A to queue B using IBM MQ.
My program works fine, but what I am concerned about is losing messages. I know that .get() removes the message from queue A. So of course, there is a brief moment where I have "got" a message from queue A, and I have not yet placed it in queue B. If my program were to crash during this time - the message would be lost.
To combat this, I am writing the current message to the logs. Then, if the program crashes, we can enter the message back into the queue manually.
However - what if the program crashes from an IOException? Now the message is gone from queue A, has not been .put() in queue B, and hasn't been written to the logs.
The way I see it, I have two options:
Browse the message first: I know that I can browse the message before "getting" it, although I'm a bit confused about how this affects the number of messages in the queue, and if it creates a duplicate etc.
Write the message back to queue A: In theory, if we "get" the message from queue A, we shouldn't have a problem "putting" it back to queue A, if for some reason we can't connect to queue B.
Could someone clarify the proper way to browse a message first - or perhaps suggest a third option I haven't thought of?
while (true) {
try {
// Clear the MQMessage
theMessage.messageId = MQConstants.MQMI_NONE;
theMessage.correlationId = MQConstants.MQCI_NONE;
// Get the message from queue A
queueA.get(theMessage, gmo);
// Read the message from queue A
byte[] messageBytes = new byte[theMessage.getMessageLength()];
theMessage.readFully(messageBytes);
String messageText = new String(messageBytes);
// Store the message to the logs in case of crash
// Put the message in queue B
queueB.put(theMessage);
} catch (MQException e) {
// Break the loop if we get an MQException
// Hopefully, it is a reason code 2033 (out of messages)
} catch (IOException e) {
// Something went wrong reading the message
}
}
Generally, if you want to keep track of messages read and written, you should use transactional reading/writing.
MQGetMessageOptions gmo = new MQGetMessageOptions();
gmo.waitInterval = 1000;
gmo.options = MQGMO_WAIT;
gmo.options += MQGMO_FAIL_IF_QUIESCING;
gmo.options += MQGMO_SYNCPOINT;
MQPutMessageOptions pmo = new MQPutMessageOptions();
pmo.options += MQPMO_SYNCPOINT;
// create message instance
MQMessage message = new MQMessage();
message.correlationId = MQCI_NONE;
message.messageId = MQMI_NONE;
// read message
queueA.get(message, gmo);
// write message
queueB.put(message, pmo);
// commit transaction
qmgr.commit();
In that case, if transaction will not be committed, all read messages will return to the source queue and all written messages will disappear from target queues. It may be good idea to commit not every message, but every 10 or 100 depending on their amount.
If you're not going to use distributed transactions (e.g. saving some information from MQ messages to database), that would suffice. Otherwise I'd recommend to switch to JMS because of its better transaction support.
Related
I'm trying to retrieve messages from a queue. I understand that RecieveMessageRequest has a threshold of 10 messages but when I tried I was able to receive only 2 out 3 messages in the queue. I read many threads which said adding setMaxNumberOfMessages(10) and increasing WaitTimeSeconds will fix it(Before adding this I received only one message out of 3) but it wasn't helpful.
FYI: I'm using a standard queue and all the messages were definitely there in the queue at the time of receive message request so it shouldn't have been a polling issue.
My implementation:
List<Message> messages;
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest().withQueueUrl(queueUrl)
.withWaitTimeSeconds(10)
.withMaxNumberOfMessages(10);
messages = sqsConfig.getSQSClient().receiveMessage(receiveMessageRequest).getMessages();
I know it make not make a ton of sense but in multiple programming languages (certainly Java and Python) I've had to loop to get everything from a queue. In Java it's something like (using the V2 api's):
// note that "done" needs to be set somewhere else to stop the loop
while (!done) {
ReceiveMessageRequest receiveMessageRequest = ReceiveMessageRequest.builder()
.queueUrl(queueUrl)
.waitTimeSeconds(20)
.build();
List<Message> messages = sqsClient.receiveMessage(receiveMessageRequest).messages();
for (Message nextMessage : messages) {
// do something with the message
DeleteMessageRequest deleteMessageRequest = DeleteMessageRequest
.builder()
.queueUrl(queueUrl)
.receiptHandle(nextMessage.receiptHandle())
.build();
sqsClient.deleteMessage(deleteMessageRequest);
}
}
From what I've read it is because there are ultimately many servers behind SQS and a single call doesn't hit all of them - it takes multiple calls. This is not a busy loop as the waitTimeSeconds does block if there is nothing to do.
Try something like this to read SQS. It's not quite as elegant but it does work.
Currently, my program is processing the messages being received from a queue but we encountered a xml file that has an error and what happens is it keeps looping on the same message and retrying to process it.
I would like to move the message to dead letter queue when a message like this occurs again.
What I did right now is that I created a class that will "producer.send(destination, msg)" to the dead queue and call this function on the try-catch but it seems that it is not working.
As #JoshMc hinted you should be treating the error messages as poison messages. For that you will need to enable transactions, and invoke a rollback for the error message.
ie. logic that looks like
// Create a connection factory
JmsFactoryFactory ff = JmsFactoryFactory.getInstance(WMQConstants.WMQ_PROVIDER);
JmsConnectionFactory cf = ff.createConnectionFactory();
//Set connection properties
...
context = cf.createContext(JMSContext.SESSION_TRANSACTED);
try {
...
// Message Processing
...
// All is OK
context.commit();
} catch (Exception e) {
// Message processing failed
context.rollback();
}
If a backout queue and backout threshold is set then the poison message is put on to the backout queue (BOQNAME) after BOTHRESH attempts at handling the message.
All this is done for you, by the underlying MQ Client code.
There is an explanation in this article - https://developer.ibm.com/articles/an-introduction-to-local-transactions-using-mq-and-jms/
which also links to sample code here - https://github.com/ibm-messaging/mq-dev-patterns/tree/master/transactions/JMS/SE
I'm trying to know when a message has been accepted (ack) or not (nack) using RabbitMQ and Spring Boot.
I want to send a message into a queue (via exchange) and check if the queue has been accepted the message. Actually I want to send to two different queues, but it is not important, I'm assuming if it works for one of them will work for the other too.
So I've tried something like this using CorrelationData:
public boolean sendMessage(...) {
CorrelationData cd = new CorrelationData();
this.rabbitTemplate.convertAndSend(exchange, routingKey, message, cd);
try {
return cd.getFuture().get(3, TimeUnit.SECONDS).isAck();
} catch (InterruptedException | ExecutionException | TimeoutException e ) {
e.printStackTrace();
return false;
}
}
The line cd.getFuture().get(3, TimeUnit.SECONDS).isAck() should get false is value has not been ack into the queue I think. But this is always true, even if routingKey doesn't exists.
So I'm assuming this piece of code is checking the message has been send into the exchange and exchange says "yes, I've recived the message, it has not been routed, but I've recived it".
So, I've looked for other ways into Rabbit/Spring documentation but I can't get the way.
And, explaining a little more, that I want is:
Into Spring Boot code I receive a message. This message has to been send to other queues/exchange, but can't be removed from the current queue (i.e. acked) until other two queues confirm the ack.
I have manual ack and as a little pseudo-code I have this:
#RabbitListener(queues = {queue})
public void receiveMessageFromDirect(Message message, Channel channel,
#Header(AmqpHeaders.DELIVERY_TAG) long tag){
boolean sendQueue1 = sendMessage(...);
boolean sendQueue2 = sendMessage(...);
if(sendQueue1 && sendQueue2){
//both messages has been readed; now I can ack this message
channel.basicAck(tag, false);
}else{
//nacked; I can't remove the message util both queue ack the message
channel.basicNack(tag,false,true);
}
I've tested this structure and, even if the queues don't exists, values sendQueue1 and sendQueue2 are always true.
The confirm is true; even for unroutable messages (I am not entirely sure why).
You need to enable returned messages (and check that it is null in the CorrelationData after the future completes - correlationData.getReturnedMessage()). If it's not null, the message wasn't routable to any queue.
You only get nacks if there is a bug in the broker, or if you are using a queue with x-max-length and overflow behavior reject-publish.
I am receiving messages from aActive MQ queue.
Is there a way to receive a number of messages in one time? or is that have to be done with a loop?
Further more, if i want to take say 30 messages run a procedure, and only if that procedure works return a message.acknowledge(); for all of them.
I mean i dont want to erase those 30 from the queue if the procedure fails.
Thanks.
You'll have to do it in a loop. Usually, it's best to use message-driven beans for consuming messages, but it's not suitable for this case, because they take message by message and you cannot specify the exact number. Thus, use MessageConsumer and manual transactions:
#Resource
UserTransaction utx;
#Resource(mappedName="jms/yourConnectionFactory");
ConnectionFactory cf;
#Resource(mappedName="jms/yourQueue");
Queue queue;
..
Connection conn = null;
Session s = null;
MessageConsumer mc = null;
try {
utx.begin();
conn = cf.createConnection();
s = conn.createSession(true, Session.CLIENT_ACKNOWLEDGE); //TRANSACTIONAL SESSION!
mc = s.createConsumer(queue);
conn.start(); // START CONNECTION'S DELIVERY OF INCOMING MESSAGES
for(int i=0; i<30; i++)
{
Message msg = mc.receive();
//BUSINESS LOGIC
}
utx.commit();
} catch(Exception ex) {
..
} finally { //CLOSE CONNECTION, SESSION AND MESSAGE CONSUMER
}
I don't have any experience in ActiveMQ. But I think in case of queue listeners, basic logic should be the same independent to the queue implementation.
For your first question I don't know any way of retrieving multiple messages from a queue. I think best way would be to fetch it one by one inside a loop.
For your second question, message will not be discarded from the queue till the underlying transaction which read the message commits. So you could read whole bunch of messages in a single transaction and roll it back in case of an error. It shouldn't erase any existing messages from the queue.
May I ask why do you need 30 messages to run a procedure. Usually when we use a queue, each message should be able to process independently.
I am using Active MQ and the Java JMS.
I want to count the number of messages on the queue.
One approach is counting the messeages with a browser:
Queue queue = (Queue) session.createQueue(subject);
QueueBrowser queueBrowser = session.createBrowser(queue);
Enumeration<?> e = queueBrowser.getEnumeration();
int numMsgs = 0;
// count number of messages
while (e.hasMoreElements()) {
// Message m = (Message) e.nextElement();
e.nextElement();
numMsgs++;
}
But for a queue with 5000 pending requests, this only return 500.
Another approach is this (iterate all the messeages in the queue):
Message message= consumer.receive(500);
while(message!= null)
{
if (message instanceof TextMessage)
{
TextMessage textMessage = (TextMessage) message;
// BytesMessage Byte
System.out.println("Received message '"+ textMessage.getText() + "'");
}
if(message!=null)
Messages_list.add(message);
message = consumer.receive(1);
}
But this also dont give the right amount of messages pending.
How can i confidently iterate akk the messages waiting in the queue?
There is a bug in ActiveMQ that is preventing the browse from returning the actual number of messages. In this case the browse is only returning a single page of messages, which is set by the maxPageSize property and documented here: http://activemq.apache.org/per-destination-policies.html
ActiveMQ currently has a bug report on this issue and it is being tracked here: https://issues.apache.org/jira/browse/AMQ-4181. This issue has been resolved and is currently scheduled to be fixed in ActiveMQ 5.8.0.
Since you are using ActiveMQ, you can make use of ActiveMQ's StatisticsPlugin: http://activemq.apache.org/statisticsplugin.html
Similarly, to query the statistics for a destination just send a message to the destination name prepended with ActiveMQ.Statistics.Destination. For example, to retrieve the statistics for a queue whose name is TEST.FOO, send an empty message to the queue named ActiveMQ.Statistics.Destination.TEST.FOO
Specifically, you might be interested in enqueueCount.
I'm omitting example code here, since the example code on the plugin's webpage are concise and good.