I'm trying to know when a message has been accepted (ack) or not (nack) using RabbitMQ and Spring Boot.
I want to send a message into a queue (via exchange) and check if the queue has been accepted the message. Actually I want to send to two different queues, but it is not important, I'm assuming if it works for one of them will work for the other too.
So I've tried something like this using CorrelationData:
public boolean sendMessage(...) {
CorrelationData cd = new CorrelationData();
this.rabbitTemplate.convertAndSend(exchange, routingKey, message, cd);
try {
return cd.getFuture().get(3, TimeUnit.SECONDS).isAck();
} catch (InterruptedException | ExecutionException | TimeoutException e ) {
e.printStackTrace();
return false;
}
}
The line cd.getFuture().get(3, TimeUnit.SECONDS).isAck() should get false is value has not been ack into the queue I think. But this is always true, even if routingKey doesn't exists.
So I'm assuming this piece of code is checking the message has been send into the exchange and exchange says "yes, I've recived the message, it has not been routed, but I've recived it".
So, I've looked for other ways into Rabbit/Spring documentation but I can't get the way.
And, explaining a little more, that I want is:
Into Spring Boot code I receive a message. This message has to been send to other queues/exchange, but can't be removed from the current queue (i.e. acked) until other two queues confirm the ack.
I have manual ack and as a little pseudo-code I have this:
#RabbitListener(queues = {queue})
public void receiveMessageFromDirect(Message message, Channel channel,
#Header(AmqpHeaders.DELIVERY_TAG) long tag){
boolean sendQueue1 = sendMessage(...);
boolean sendQueue2 = sendMessage(...);
if(sendQueue1 && sendQueue2){
//both messages has been readed; now I can ack this message
channel.basicAck(tag, false);
}else{
//nacked; I can't remove the message util both queue ack the message
channel.basicNack(tag,false,true);
}
I've tested this structure and, even if the queues don't exists, values sendQueue1 and sendQueue2 are always true.
The confirm is true; even for unroutable messages (I am not entirely sure why).
You need to enable returned messages (and check that it is null in the CorrelationData after the future completes - correlationData.getReturnedMessage()). If it's not null, the message wasn't routable to any queue.
You only get nacks if there is a bug in the broker, or if you are using a queue with x-max-length and overflow behavior reject-publish.
Related
I've read Netty Guide, it does not explain much on ChannelFuture. I find ChannelFuture is a complex idea when applying it.
What I am trying to do is to write message to a context after it's initial response. Different from typical request/response flow. I need a flow like this:
Client send request -> Server (netty)
Server send a response with ctx.writeAndFlush(msg);
Server send some more message to that ctx after step 2 is complete.
The problem is that if I do something like this, the second write will not send out:
ctx.writeAndFlush(response);
Message newMsg = createMessage();
ctx.writeAndFlush(newMsg); //will not send to client
Then I try to use ChannelFuture, it works, but I am not sure if I am logically correct:
ChannelFuture msgIsSent = ctx.writeAndFlush(response);
if(msgIsSent.isDone())
{
Message newMsg = createMessage();
ctx.writeAndFlush(newMsg); //this works
}
or should I use a ChannelFutureListener() instead?
ChannelFuture msgIsSent = ctx.writeAndFlush(response);
msgIsSent.addListener(new ChannelFutureListener(){
#Override
public void operationComplete(ChannelFuture future)
{
Message newMsg = createMessage();
ctx.writeAndFlush(newMsg);
}
});
Will this also works?
Which one is the best practice approach? Is there any potential problem using method 2?
Of course, this depends too on your "protocol" (meaning for instance if you use HTTP, sending 2 answears for the same request is not supported by HTTP protocol). But let say your protocol allows you to send multiple response parts:
Netty add messages to send to the pipeline, respecting the order.
So in your first example, I'm a bit surprised it does not work:
ctx.writeAndFlush(response);
Message newMsg = createMessage();
ctx.writeAndFlush(newMsg); // should send the message
However it could be lead by your protocol. For instance, this could happen:
response in message queue to send
flush not yet done
newMsg in message queue to send
flush now come but protocol does not support 2 messages so only send first one
So if your protocol must admit that first message is sent already, then you have to wait for the first, so doing something like:
ctx.writeAndFlush(response).addListener(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture future) {
if (future.isDone()) {
Message newMsg = createMessage();
ctx.writeAndFlush(newMsg);
} else { // an error occurs, do perhaps something else
}
}
});
So your last proposal (I've just don't create a ChannelFuture but directly used the result of writeAndFlush, but both are equals). Just take care of the case where operationComplete does not mean it is in success.
Try this:
ctx.channel().writeAndFlush(response);
Message newMsg = createMessage();
ctx.channel().writeAndFlush(newMsg);
Channel.write() always starts from the tail of the ChannelPipeline.
ChannelHandlerContext.write() starts from the current position of the ChannelHandler.
#2 looks better but make sure to test if the operation was successful. If not, use future.getCause() to access the exception. Not that it will change the functionality, but you can shorten the code by simply adding the listener directly on the result of the write call, I.e. you don't need to declare the future itself since it will be provided in the callback.
I have the following listener method:
#Override
public void onMessage(Message message, Channel channel) {
try {
// do something bad :)
} catch (Exception e){
try {
long dt = null != message.getMessageProperties()
? message.getMessageProperties().getDeliveryTag()
: 0;
channel.basicReject(dt, true);
} catch(IOException io) {
logger.error("IO-COMMON", io);
}
}
}
The issue is basic reject doesn't work, I don't know why. How to reject it gracefully? I think that if I reject a message, it should be requeued and reside is sth like cache, before going to next worker. But in fact this message just seems to be lost.
You need to set the acknowledgemode to MANUAL if you are doing your own acks. I am not sure why it's not working for you; DEBUG/TRACE logging might help.
You should consider letting the container handle the acks - use acknowledgemode=AUTO; the container will normally requeue the message for any exception thrown or ack it if the listener returns normally.
You can set defaultRequeueRejected to false (it is true by default) and the message will be discarded (or routed to a DLX/DLQ).
You can also throw an AmqpRejectAndDontRequeueException to override the default mechanism of requeuing failed messages.
If the ack mode is NONE - there are no acks and RabbitMQ automatically acks the message as soon as it's sent.
I am trying to fetch a message with a particular correlation id like explained in rabbitmq docs. However I see that the irrelevant messages gets dequeued. I do not want it to happen. How can I tell rabbitmq to not dequeue after I get message and get to know that this is not the one I was looking for. Please help me.
`
.
.
replyQueueName = channel.queueDeclare().getQueue();
consumer = new QueueingConsumer(channel);
channel.basicConsume(replyQueueName, false, consumer);
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
System.out.println(delivery.getProperties().getCorrelationId());
if (delivery.getProperties().getCorrelationId().equals(corrId)) {
response = new String(delivery.getBody());
break;
}
}
`
You can't do what you want, the way you want. The "selective consumer" is an anti-pattern in RabbitMQ.
Instead, you should design your RabbitMQ setup so that your messages are routed to a queue that only contains messages for the intended consumer.
I wrote more about this, here: http://derickbailey.com/2015/07/22/airport-baggage-claims-selective-consumers-and-rabbitmq-anti-patterns/
If you can afford to lose the order of messages you can use the re-queueing mechanism.
Try turning off auto ack.
If not, you have to redesign your application to inject headers or routing keys to route to a particular queue.
In reference to this article I think I know the difference between QoS1 and QoS2 messages, but I don't know the difference in handling both of them as a Paho MQTT client.
Imagine I subscribe to the topic like this:
MqttClient subscriber = new MqttClient(SERVER_URI, SUBSCRIBER_ID);
subscriber.subscribe(TOPIC);
And then I'm publishing messages like this:
publisher.publish(TOPIC, PAYLOAD, 1, false);
At this moment I'm using MqttCallback interface to handle messages that arrives to subscribers.
There is a method to override:
public void messageArrived(String topic, MqttMessage mqttMessage) {
if(mqttMessage.isDuplicate()) {
// is it really the duplicate message from my perspective?
} else {...}
}
In the MqttMessage we can find a isDuplicate() method, but how can I be sure that the mqttMessage that returns true is not the first message that my subscriber received?
I am very interested in finding a solution that shows how to handle QoS1, but every answer which will clarify anything here will be appreciated.
Best regards from Cracow!
It's not sufficient to rely on the duplicate flag, since you could have missed the first message. If the QoS 1 messages are not idempotent, here are a few suggestions how you could do duplicate detection:
Hash the payload + topic and have a table with the last X messages and their hashes available so you can check if you already received that message
Have a unique ID in the payload and have a table with the last X messages and their ID available
Have a timestamp in the payload and have a table with the last X messages and their timestamp available
If you really need to make sure that a message arrives once and only once, you can use QoS 2. QoS 1 means that your clients can handle duplicates (either by ignoring a duplicate message or the messages are idempotent).
I encountered an interresting and I think very common synchronization problem in my test code.
This is the test (its a functional test that connects from the outside to the system), i run it via TestNG.
#Test
public void operationalClientConnected_sendGetUserSessionRequest_clientShallReceiveGetUserSessionResponse() {
// GIVEN
OperationalClientSimulator client = operationalClientHasEstablishedWebSocketConnection("ClientXY");
// WHEN
GetUserSessionRequest request = PojoRequestBuilder.newRequest(GetUserSessionRequest.class).build();
client.sendRequest(request);
// THEN
assertThatClientReceivesResponse(client, GetUserSessionResponse.class, request.getCorrelationId(), request.getRequestId());
}
Basically i send a single request and wait for the correct response, this is what i want to verify in this test.
Behind the assertThatClientReceivesResponse there is a hamcrest matcher that looks like this:
#Override
protected boolean matchesSafely(final OperationalClientSimulator client) {
Object awaitedMessage = client.awaitMessage(
new Verification<Object>() {
#Override
public VerificationResult verify(final Object actual) {
VerificationResult result = new VerificationResult();
if (!_expectedResponseClass.isInstance(actual)) {
result.addMismatch("not of expected type", actual, _expectedResponseClass.getSimpleName());
}
// check more details of message ..
return result;
}
}, _expectedTimeout);
boolean matches = awaitedMessage != null;
if (matches) {
_messageCaptor.setActualMessage((T) awaitedMessage);
}
return matches;
}
Now to the interresting part, the synchronization in the OperationalClientSimulator class.
Two methods are of interrest:
awaitMessage which blocks until either a message that matches the given Verification is received or the timeout expired
onMessage received method which is called for each message that is received (over a websocket connection)
Basically what I want to achive is having the test thread block on the awaitMessage method until either the correct message is received (via onMessage) or the specified timeout elapsed.
public Object awaitMessage(final Verification<Object> verification, final long timeoutMillis) {
// howto sync?
return awaitedMessage; // or null
}
#Override
public void onMessage(final String message) {
LOG.info("#Client {} <== received a message on websocket - {}", name, message);
// howto sync?
}
About the test:
The test thread will almost always be faster and therefor has to wait until the response is received via the awaitMessage method
There can be very rare cases when the expected message is received before the test thread is checking for it (this basically means i have to save every received message)
In this specific test case there are only a handfull of messages that are received (some heartbeat messages, the actual response and a notification), but in other cases there can be hundreds of messages which in need to inspect to find the expected message(s)
I was thinking about different solutions for synchronizing here:
The simplest of course would be the sync with the synchronized keyword but I think there are neater ways to do this
The onMessage received method could simply write into a blocking queue and the test thread can consume from it but here I dont know how to measure the timeout.. can I use a CountdownLatch?
Maybe I can do a non blocking solution where the producer (onMessage) writes into an Array and the consumer reads until it reaches an index that is published by the producer (like the LMAX Disruptor)
I know this is test code and performance is not really an issue here, I am just thinking how to solve this in a "nice" way.. you know.. because its christmas :-)
So the actual question here is, how do i "safely" wait for the message which i expect in my test with a timeout? Safely here means that i never miss a message or lose a message because of concurrency issues and that I also need to check if the expected message was already received.
How should I synchronize between the test runner thread and the thread that calls the onMessage method in the OperationalClientSimulator when a message is received on the websocket connection.