Develop Reference

java-nats-streaming: Publishing Messages after Server Reconnects

I have a NATS streaming cluster with 3 nodes set up. It seems that NATS messages published by my java application during server downtime is lost (i.e. not republished again when my servers are back up and running).
A more detailed description:
NATS cluster online. Publisher and Subscriber applications come online. Publisher begins to publish a message every second. Subscriber receives messages.
NATS servers are shut off. Publisher continues to publish messages (let's call these messages 'offline messages'). Subscriber stops receiving anything
NATS servers come back online. Subscriber begins to receive messages again, but 'offline messages' are never received.
Both my publisher and subscriber applications are configured to attempt reconnection to NATS server and does not timeout. I do not get any exceptions throughout.
NATS connection:
Options options = new Options.Builder().servers(serverList).maxReconnects(-1).build();
Connection nc = Nats.connect(options);
StreamingConnectionFactory cf = new StreamingConnectionFactory(natsProperties.getClusterId(), natsProperties.getClientId());
cf.setNatsConnection(nc);
streamingConnection = cf.createConnection();
Publisher:
// subject and message String variables are passed in
streamingConnection.publish(subject, message.getBytes());
Subscriber:
streamingConnection.subscribe(subject, new MessageHandler() {
public void onMessage(Message m) {
System.out.prinf("Received msg: %s\n", m.getData())
}
}, new SubscriptionOptions.Builder().durableName(durableName).build());
From the docs, the Java NATS client seems to have a reconnect buffer built in. I tried increasing the buffer by a factor of 10 but to no avail (also, my messages consist only of 2-digit numbers). How do I get it to resend these 'offline messages'?

I have the same problem, the only solution that I see that another method of subscription is occupied, save the sequence of messages but this I do not think is the best
// Receive messages starting at a specific sequence number
sc.subscribe("foo", new MessageHandler() {
public void onMessage(Message m) {
logger.info("Sequence message " + m.getSequence());
System.out.printf("Received a message: %s\n", m.getData());
}
}, new SubscriptionOptions.Builder().startAtSequence(22).build());

How to browse the queue in rabbitmq without dequeuing the messages

I am trying to fetch a message with a particular correlation id like explained in rabbitmq docs. However I see that the irrelevant messages gets dequeued. I do not want it to happen. How can I tell rabbitmq to not dequeue after I get message and get to know that this is not the one I was looking for. Please help me.
`
.
.
replyQueueName = channel.queueDeclare().getQueue();
consumer = new QueueingConsumer(channel);
channel.basicConsume(replyQueueName, false, consumer);
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
System.out.println(delivery.getProperties().getCorrelationId());
if (delivery.getProperties().getCorrelationId().equals(corrId)) {
response = new String(delivery.getBody());
break;
}
}
`

You can't do what you want, the way you want. The "selective consumer" is an anti-pattern in RabbitMQ.
Instead, you should design your RabbitMQ setup so that your messages are routed to a queue that only contains messages for the intended consumer.
I wrote more about this, here: http://derickbailey.com/2015/07/22/airport-baggage-claims-selective-consumers-and-rabbitmq-anti-patterns/

If you can afford to lose the order of messages you can use the re-queueing mechanism.
Try turning off auto ack.
If not, you have to redesign your application to inject headers or routing keys to route to a particular queue.

QoS1 vs QoS2 messages - difference in handling it with Paho MQTT client in Java

In reference to this article I think I know the difference between QoS1 and QoS2 messages, but I don't know the difference in handling both of them as a Paho MQTT client.
Imagine I subscribe to the topic like this:
MqttClient subscriber = new MqttClient(SERVER_URI, SUBSCRIBER_ID);
subscriber.subscribe(TOPIC);
And then I'm publishing messages like this:
publisher.publish(TOPIC, PAYLOAD, 1, false);
At this moment I'm using MqttCallback interface to handle messages that arrives to subscribers.
There is a method to override:
public void messageArrived(String topic, MqttMessage mqttMessage) {
if(mqttMessage.isDuplicate()) {
// is it really the duplicate message from my perspective?
} else {...}
}
In the MqttMessage we can find a isDuplicate() method, but how can I be sure that the mqttMessage that returns true is not the first message that my subscriber received?
I am very interested in finding a solution that shows how to handle QoS1, but every answer which will clarify anything here will be appreciated.
Best regards from Cracow!

It's not sufficient to rely on the duplicate flag, since you could have missed the first message. If the QoS 1 messages are not idempotent, here are a few suggestions how you could do duplicate detection:
Hash the payload + topic and have a table with the last X messages and their hashes available so you can check if you already received that message
Have a unique ID in the payload and have a table with the last X messages and their ID available
Have a timestamp in the payload and have a table with the last X messages and their timestamp available
If you really need to make sure that a message arrives once and only once, you can use QoS 2. QoS 1 means that your clients can handle duplicates (either by ignoring a duplicate message or the messages are idempotent).

Subscribe to multiple topic destination in ActiveMQ Messaging

A publisher publishes messages to different destinations. My client needs to subscribe and get all those messages in those destinations one by one.Means i want to consume messages from multiple topics. Also I want the topic messages (different destinations) to be received in a button action, not by using Message Listener. Can anyone please help on this?
Part of my code is.
MessageConsumer consumer = null;
if (isDurableSubscription) {
// the subscription Name assigned to a durable subscription must be unique within a given client ID.
consumer = session.createDurableSubscriber( topic, subscriptionName );
} else {
consumer = session.createConsumer( topic );
}
log.finest("consumer = " + consumer );
consumer.setMessageListener( this );
conn.start();
}
public void onMessage(Message message) {
if ( message instanceof TextMessage ) {
try {
TextMessage txtMessage = (TextMessage) message;
String text = txtMessage.getText();
this.msg = text;
System.out.println(text);
log.finest("Message processed ...");
session.commit();
}

Also i want the topic messages (different destinations) to be
received in a button action, not by using Message Listener.
The whole point of a JMS provider is to listen to messages published by a producer and have an async communication channel in which the producer and the listener are decoupled. When you say you want to receive messages in a button action, it's equivalent of saying "I don't really care when the publisher produced the message, but I'll listen when I feel like" - which doesn't fit the use of a JMS. May be a queue where you have messages and pick one after the other based on some user action.
The publisher will not mark the message as delivered (based on how you have configured it) until the client acknowledges it and in your case (even if it were possible), it may be a long time and the message might expire. One way to achieve this, with JMS, is to have your internal data structure where you keep all your messages (after picking them up from the topic using a listener) and then process it on a button action. But you'll lose all the benefits of a JMS provider (durability, loss of messages upon client shut down, and the likes).

Mqtt Paho - Trying to publish while broker is unreachable

I'm begininng to use Mqtt and I have a hard time with handling an unreliable network.
I'm using a Paho Java Client (in groovy) to publish messages to a distant Mosquitto Broker.
Is there a way, when the broker is unreachable, to have the Paho client persist the message and automatically re-connect to the broker and publish the locally stored messages ? Do I have to handle everything myself, using for example a local broker ?
Here is my client building code
String persistenceDir = config['persistence-dir'] ?: System.getProperty('java.io.tmpdir')
def persistence = new MqttDefaultFilePersistence(persistenceDir)
client = new MqttAsyncClient(uri, clientId, persistence)
client.setCallback(this)
options = new MqttConnectOptions()
if (config.password) {
options.setPassword(config.password as char[])
options.setUserName(config.user)
}
options.setCleanSession(false)
client.connect(options)
And my publish code
def message = new MqttMessage(Json.encode(outgoingMessage).getBytes())
try {
client?.connect(options)
def topic = client.getTopic('processMsg')
message.setQos(1)
def token = topic.publish(message)
if (client) {
client.disconnect()
}
Thanks

The Paho client will only persist in-flight messages when it is connected to the broker.
Typically, when connectivity issues start to arrive you'll see message timeouts popping up
Timed out waiting for a response from the server (32000)
At that point the message will still be persisted.
However, when the connection is lost, and you start seeing this
Client is not connected (32104)
You should assume that the message has not been persisted by Paho.
You can debug this in org.eclipse.paho.client.mqttv3.internal.ClientComms :
/**
* Sends a message to the broker if in connected state, but only waits for the message to be
* stored, before returning.
*/
public void sendNoWait(MqttWireMessage message, MqttToken token) throws MqttException {
final String methodName = "sendNoWait";
if (isConnected() ||
(!isConnected() && message instanceof MqttConnect) ||
(isDisconnecting() && message instanceof MqttDisconnect)) {
this.internalSend(message, token);
} else {
//#TRACE 208=failed: not connected
log.fine(className, methodName, "208");
throw ExceptionHelper.createMqttException(MqttException.REASON_CODE_CLIENT_NOT_CONNECTED);
}
}
The internalSend will persist the message, but only if it is connected to the broker.
Also take into account that there is a maximum number of inflight messages that Paho can process. If it exceeds that it will also decide to not persist the message.

You could just setup a local broker and bridge that with the remote broker. That way you can queue up all your messages locally and when the remote broker comes back online all can be delivered.

Yes... After you get an exception that the message can't be delivered, it has to be either persisted or the message needs to be regenerated.
If you plan to use a local broker you can look at Really Small Message Broker (https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=d5bedadd-e46f-4c97-af89-22d65ffee070)