I try to use Kafka Java SDK to implement a consumer however most consumer examples I saw are using while(true) loop and inside the loop call consume method to get a message.
while (true) {
final ConsumerRecords<Long, String> consumerRecords =
consumer.poll(1000);
if (consumerRecords.count()==0) {
noRecordsCount++;
if (noRecordsCount > giveUp) break;
else continue;
}
consumerRecords.forEach(record -> {
System.out.printf("Consumer Record:(%d, %s, %d, %d)\n",
record.key(), record.value(),
record.partition(), record.offset());
});
consumer.commitAsync();
}
I am wondering are there any elegant way to handle this without using while loop which is similar to RabbitMQ implementation following:
Consumer consumer = new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body)
throws IOException {
String message = new String(body, "UTF-8");
System.out.println(" [x] Received '" + message + "'");
}
};
channel.basicConsume(QUEUE_NAME, true, consumer);
You can try using Spring-kafka which has #KafkaListener annotation and makes the method to listen topic, for more info here
Because in apache-kafka there is no elegant way to make method as a listener to topic, since consumer need to poll records for certain intervals, needed that code in loop
#KafkaListener(topics = "topicName", group = "foo")
public void listen(String message) {
System.out.println("Received Messasge in group foo: " + message);
}
Poll Loop is the only way to consume the messages in Kafka. The elegant code to handle the message should be within the loop.
Related
Receive single message from the queue in RabbitMQ using java:
I'm new to RabbitMQ and was wondering of a good approach to this problem I'm mulling over. I want to create a service that subscribes to a queue and Receive only one message and receive the next one after processing.
DeliverCallback deliverCallback = new DeliverCallback() {
public void handle(String s, Delivery delivery) throws IOException {
System.out.println("Tag: "+delivery.getEnvelope().getDeliveryTag());
String message = new String(delivery.getBody(), "UTF-8");
}
};
channel.basicConsume(QUEUE_NAME, false, deliverCallback, new CancelCallback() {
public void handle(String consumerTag) throws IOException {}
});
To receive a single message from the queue, I did the following steps:
Step 1: Accept only one unack-ed message at a time:
channel.basicQos(1);
Step 2: Setting autoAck to false:
boolean autoAck = false;
Step 3: Work Queues using the Java Client
DeliverCallback deliverCallback = (consumerTag, delivery) -> {
try {
System.out.println("Tag: "+delivery.getEnvelope().getDeliveryTag());
String message = new String(delivery.getBody(), "UTF-8");
} finally {
System.out.println(" [Message] Task Done");
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
}
};
channel.basicConsume(QUEUE_NAME, autoAck, deliverCallback, consumerTag -> { });
see: rabbitmq documentation
RabbitMQ supports polling a single message from a Queue at a time, but the official document doesnt encourage this approach.
Poll the message by disabling auto acknowledgement, finally send an ack after processing.
boolean autoAck = false;
GetResponse response = channel.basicGet(queueName, autoAck);
if (response != null) {
//process the message and acknowledge it
}
For more details please refer to the official document section 'Retrieving Individual Messages ("Pull API")'
https://www.rabbitmq.com/api-guide.html#getting
This is a very contrived example of a rabbitmq app which is both producer and consumer of messages in one single main method. The problem is the code inside overridden handleDelivery method never gets executed. I use Rabbitmq dashboard and see the queue fills up and consumes. And the line in handleConsumeOk gets printed.
Since I am new to rabbitmq I'm wondering if I have done something fundamentally wrong or I just got the idea of "Called when a basic.deliver is received for this consumer" wrong.
public class RabbitMain {
public static void main(String[] args) throws IOException, TimeoutException, InterruptedException {
Connection connection = Utils.getConnection("test");
String payload = "hello world!";
try (Channel channel = connection.createChannel()){
channel.exchangeDeclare("sampleExchange", BuiltinExchangeType.TOPIC, true);
channel.basicPublish("sampleExchange", "testKey", null, payload.getBytes());
}
System.out.println("Consume...");
try (Channel channel = connection.createChannel()){
channel.exchangeDeclare("sampleExchange", BuiltinExchangeType.TOPIC, true);
channel.queueDeclare("testQueue", true, false, false, null);
channel.queueBind("testQueue", "sampleExchange", "testKey");
Consumer consumer = new DefaultConsumer(channel){
#Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
String message = new String(body);
System.out.println("Received: " + message);
}
#Override
public void handleConsumeOk(String consumerTag) {
System.out.println("handled consume ok");
}
};
Thread.sleep(2000);
channel.basicConsume("testQueue", true, consumer);
}
}
}
You publish the message to the exchnage before any queue is bound to it. RabbitMQ will discard any message it can't route to a queue.
Do
channel.queueDeclare("testQueue", true, false, false, null);
channel.queueBind("testQueue", "sampleExchange", "testKey");
before you call channel.basicPublish.
Problem statement:- I am trying to automate a MQTT flow, for that I a need to publish and subscribe to multiple topics but in a sequential order. The trick part is that the message received from the first publish has some value which will be passed in the next sub/pub commands.
For eg.
Sub to topicA/abc
Pub to topicA/abc
Message received on topicA/abc is xyz
sub to topic topicA/xyz
pub to topic topicA/xyz
I am able to receive the message on the first topic but I am not getting how to access the payload of the received message in the main method and pass and attach it to the next topic for next sub.
Is there a way to get the retrieved the message payload from messageArrived callback method to the main method where is client instance is created?
Note:- I am using a single client for publish and subscribe.
kindly help me out as I have ran out of options and methods to do so.
Edited:-
Code snippet
Main class
public class MqttOverSSL {
String deviceId;
MqttClient client = null;
public MqttOverSSL() {
}
public MqttOverSSL(String deviceId) throws MqttException, InterruptedException {
this.deviceId = deviceId;
MqttConnection mqttConObj = new MqttConnection();
this.client = mqttConObj.mqttConnection();
}
public void getLinkCodeMethod() throws MqttException, InterruptedException {
client.subscribe("abc/multi/" + deviceId + "/linkcode", 0);
publish(client, "abc/multi/" + deviceId + "/getlinkcode", 0, "".getBytes());
}
}
Mqtt Claback impl:-
public class SimpleMqttCallBack implements MqttCallback {
String arrivedMessage;
#Override
public void connectionLost(Throwable throwable) {
System.out.println("Connection to MQTT broker lost!");
}
#Override
public void messageArrived(String s, MqttMessage mqttMessage) throws Exception {
arrivedMessage = mqttMessage.toString();
System.out.println("Message received:\t" + arrivedMessage);
linkCode(arrivedMessage);
}
#Override
public void deliveryComplete(IMqttDeliveryToken iMqttDeliveryToken) {
System.out.println("Delivery complete callback: Publish Completed "+ Arrays.toString(iMqttDeliveryToken.getTopics()));
}
public void linkCode(String arrivedMessage) throws MqttException {
System.out.println("String is "+ arrivedMessage);
Gson g = new Gson();
GetCode code = g.fromJson(arrivedMessage, GetCode.class);
System.out.println(code.getLinkCode());
}
}
Publisher class:-
public class Publisher {
public static void publish(MqttClient client, String topicName, int qos, byte[] payload) throws MqttException {
String time = new Timestamp(System.currentTimeMillis()).toString();
log("Publishing at: "+time+ " to topic \""+topicName+"\" qos "+qos);
// Create and configure a message
MqttMessage message = new MqttMessage(payload);
message.setQos(qos);
// Send the message to the server, control is not returned until
// it has been delivered to the server meeting the specified
// quality of service.
client.publish(topicName, message);
}
static private void log(String message) {
boolean quietMode = false;
if (!quietMode) {
System.out.println(message);
}
}
}
OK, it's a little clearer what you are trying to do now.
Short answer No, you can not pass values back to the "main method". MQTT is asynchronous that means you have no idea when a message will arrive for a topic you subscribe to.
You need to update your code to deal check what the incoming message topic is and then deal do what ever action you wanted to do with that response in the messageArrived() handler. If you have a sequence of task to do then you may need to implement what is known as a state machine in order to keep track of where you are in the sequence.
How to read a message from WebSphere MQ without deleting the original message from queue?
I have spring application which reads the message from the WebSphere MQ.
After reading, I have a process method which will process the data retrieved from queue.
Step 1:
response = jmsTemplate.receive();
//Message automatically removed from queue.
Step 2:
process(response);
There are chances of throwing exceptions in process method. In case of exceptions, I need to retain the message in the queue.
Is it possible? Is their any way to delete the message only on user acknowledgement?
I tried adding the following:
jmsTemplate.setSessionAcknowledgeMode(javax.jms.Session.CLIENT_ACKNOWLEDGE);
...but still the message is getting deleted.
JmsTemplate creating code snippet:
JndiConnectionFactorySupport connectionFactoryBean = new JndiConnectionFactorySupport();
connectionFactoryBean.setBindingsDir(this.bindingDir);
connectionFactoryBean
.setConnectionFactoryName(connectionFactoryName);
connectionFactoryBean.afterPropertiesSet();
jmsTemplate.setConnectionFactory(connectionFactoryBean.getObject());
JndiDestinationResolver destinationResolver = new JndiDestinationResolver();
destinationResolver.setJndiTemplate(connectionFactoryBean
.getJndiTemplate());
jmsTemplate.setDestinationResolver(destinationResolver);
jmsTemplate.setReceiveTimeout(20000);
jmsTemplate.setDefaultDestinationName(this.defaultDestinationName);
Tried the jmsTemplate.execute() method as below:
#SuppressWarnings({ "unused", "unchecked" })
Message responseMessage = (Message) jmsTemplate.execute(
new SessionCallback() {
public Object doInJms(Session session)
throws JMSException {
MessageConsumer consumer = session
.createConsumer(jmsTemplate.getDestinationResolver().resolveDestinationName(session, "QUEUE_NAME", false));
Message response = consumer.receive(1);
try {
testMethod();//this method will throw exception.
response.acknowledge();
consumer.close();
} catch(Exception e){
consumer.close();//control will come here.
}
return response;
}
}, true);
You can't do that with receive() methods because the operation is complete (from the session perspective) when the receive method returns.
You need to run the code that might fail within the scope of the session; e.g. with a JmsTemplate.execute() with a SessionCallback - something like this...
this.jmsTemplate.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
this.jmsTemplate.convertAndSend("foo", "bar");
try {
String value = this.jmsTemplate.execute(session -> {
MessageConsumer consumer = session.createConsumer(
this.jmsTemplate.getDestinationResolver().resolveDestinationName(session, "foo", false));
String result;
try {
Message received = consumer.receive(5000);
result = (String) this.jmsTemplate.getMessageConverter().fromMessage(received);
// Do some stuff that might throw an exception
received.acknowledge();
}
finally {
consumer.close();
}
return result;
}, true);
System.out.println(value);
}
catch (Exception e) {
e.printStackTrace();
}
You have to browse the queue.
Example of real code that was executed making usage of Websphere MQ
public void browseMessagesAndJiraCreation(String jiraUserName, String jiraPassword) {
int counterMessages = jmsTemplate.browse(destinationQueueName, new BrowserCallback<Integer>() {
#Override
public Integer doInJms(final Session session, final QueueBrowser queueBrowser) throws JMSException {
Enumeration<TextMessage> enumeration = queueBrowser.getEnumeration();
int counterMessages = 0;
while (enumeration.hasMoreElements()) {
counterMessages += 1;
TextMessage msg = enumeration.nextElement();
logger.info("Found : {}", msg.getText());
JiraId jiraId = jiraManager.createIssue(jiraUserName, jiraPassword);
jiraManager.attachFileToJira(jiraId, msg.getText(), jiraUserName, jiraPassword);
}
return counterMessages;
}
});
logger.info("{}:messages were browsed and processed from queue:{}.", counterMessages, destinationQueueName);
}
Explanations:
usage of the Spring Framework JmsTemplate
you pass the String gestinationQueueName (example destinationQueueName=QL.PREFCNTR.USER.REPLY)
Java enumeration of Text messages
counterMessages is the counter of messages that were processed
messages are NOT consumed!
You can add transactional processing of JMS messages. See the example
Your listener should be "transacted". Like this
<jms:listener-container connection-factory="connectionFactory" acknowledge="transacted">
<jms:listener ref="notificationProcessor" destination="incoming.queue"/>
</jms:listener-container>
I've recently learned RabbitMQ with hopes of implementing it in my work flow. (I will be implementing it in Java) I just finished all the tutorials and was curious how I would implement a "constant" queue instead of a "temporary" queue. Or at least allow the consumer to get the message that the exchange sent. For example if I send a topic of "kern.overflow" but a consumer is offline, as soon as my consumer comes online as long as it is listening for something related to "kern.#" or "#.overflow" I want it to receive un-received messages.
Here is the code to:
create a persistent queue
bind the queue to the topic with "kern.#" as routing-key:
code:
String myPersistentQueue = "myPersistentQueue";
boolean isDurable = true;
boolean isExclusive = false;
boolean isAutoDelete = false;
channel.queueDeclare(myPersistentQueue, isDurable, isExclusive, isAutoDelete, null);
channel.queueBind(myPersistentQueue, "myTopic", "kern.#");
final QueueingConsumer consumer = new QueueingConsumer(channel);
boolean autoAck = true;
String tag1 = channel.basicConsume(myPersistentQueue, autoAck, consumer);
executorService.execute(new Runnable() {
#Override
public void run() {
while (true) {
Delivery delivery;
try {
delivery = consumer.nextDelivery();
String message = new String(delivery.getBody());
System.out.println("Received: " + message);
} catch (Exception ex) {
Logger.getLogger(TestMng.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
});
System.out.println("Consumers Ready");
When you publish a message to myTopic using kern.overflow as routing-key the message is stored to the myPersistentQueue queue. The client can be off-line, when the client is on-line can get the messages.