Develop Reference

How to commit after each kafka message in spring-kafka?

In our application we use a kafka consumer to determine to send an email.
There was an issue we had the other day, where the kafka partition was timing out before it could read and process all its records. As a result, it looped back to the start of the partition and wasn't able to finished the set of records it had received and new data generated after the loop start never got processed.
My teams suggested that we could tell Kafka to commit after each message is read, however I can't figure out how to do that from Spring-kakfa.
The application uses spring-kafka 2.1.6, and the consumer code kinda resembles this.
#KafkaListener(topics = "${kafka.topic}", groupId = "${kafka.groupId}")
public void consume(String message, #Header("kafka_offset") int offSet) {
try{
EmailData data = objectMapper.readValue(message, EmailData.class);
if(isEligableForEmail(data)){
emailHandler.sendEmail(data)
}
} catch (Exception e) {
log.error("Error: "+e.getMessage(), e);
}
}
Note: sendEmail function uses CompletableFutures, as it has to call a different API before sending out an email.
Configuration: (snippet of the yaml file for the consumer, and a part of the producer)
consumer:
max.poll.interval.ms: 3600000
producer:
retries: 0
batch-size: 100000
acks: 0
buffer-memory: 33554432
request.timeout.ms: 60000
linger.ms: 10
max.block.ms: 5000

If you want manual Acknowledgment, then you can provide the Acknowledgment in method arguments
#KafkaListener(topics = "${kafka.topic}", groupId = "${kafka.groupId}")
public void consume(String message, Acknowledgment ack, #Header("kafka_offset") int offSet) {
When using manual AckMode, you can also provide the listener with the Acknowledgment. The following example also shows how to use a different container factory.
Example from docs #KafkaListener Annotation
#KafkaListener(id = "cat", topics = "myTopic",
containerFactory = "kafkaManualAckListenerContainerFactory")
public void listen(String data, Acknowledgment ack) {
...
ack.acknowledge();
}

Set the container ackMode property to AckMode.RECORD to commit the offset after each record.
You should also consider reducing max.poll.records or increasing max.poll.interval.ms Kafka consumer properties.

How to get the current number of messages in a jms topic on IBM WAS

I need to create a REST service in the Java programming language that receives JMS connection factory's JNDI name and JMS topic's JNDI name as input and should return the number of messages in the resource at the moment.
The problem is getting the length of the topic from IBM WAS.
I know about the existence of TopicBrowser from Oracle, with which you can get all the messages in the topic and just count their number. But for some reason, we do not use it.
My idea is to get the SIB Destination queue length property, which is located in:
Buses> Bus> Recipients> MyTopic.Space> Publication Points, in the web console.
I use:
IBM WAS 9.0. ND.
Default Message Provider.
I will be glad to any advice.

You can get the state of a subscriber using the admin client for example:
// Start by querying the objectName of the Publication Point (Topic Space on a specific node).
AdminClient adminClient = AdminClientFactory.createAdminClient(connectProps);
StringBuffer oNameQuery= new StringBuffer();
oNameQuery.append(adminClient.getDomainName()).append(":*");
oNameQuery.append(",type=").append("SIBPublicationPoint");
oNameQuery.append(",name=").append("Default.Topic.Space");
oNameQuery.append(",node=").append(nodeName);
oNameQuery.append(",process=").append("server1");
oSet= adminClient.queryNames(new ObjectName(oNameQuery.toString()), null);
ObjectName defaultTopicSpaceOn = (ObjectName) oSet.iterator().next();
System.out.println("Default.Topic.Space ObjectName:"+defaultTopicSpaceOn);
// Then look at each subscription storing messages in the Publication Point.
Long depth = (Long) adminClient.invoke (defaultTopicSpaceOn, "getDepth", null, null);
System.out.println("DefaultTopicSpace Depth:"+depth+"\n");
SIBSubscription[] subscriptions = (SIBSubscription[]) adminClient.invoke (defaultTopicSpaceOn, "getSubscriptions", null, null);
for (SIBSubscription subscription : subscriptions) {
System.out.print("DefaultTopicSpace Subscription:"+subscription.getName()
+" Id:"+subscription.getId()
+" SubscriberId:"+subscription.getSubscriberId()
+" Selector:"+subscription.getSelector()
+" Depth:"+subscription.getDepth());
for (String topicName: subscription.getTopics())
System.out.print(" Topic:"+topicName);
System.out.println();
}
This produces something like:
DefaultTopicSpace Depth:2
DefaultTopicSpace Subscription:Default.Topic.Space Id:21974964F5B726A6C21C7E59 SubscriberId:jmsThinClient.JMSSendReceiveclientID##jmsThinClient.JMSSendReceiveSubscription Selector:null Depth:2 Topic:Topic1/*
Api doc:https://www.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.javadoc.doc/web/apidocs/com/ibm/websphere/sib/admin/package-summary.html

How to retrieve details of all broker from kafka cluster

How should we retrieve the full details of all brokers(connected/disconnected) from kafka cluster/zookeeper ?
I found following way to fetch only active brokers, but I want the the IP address of broker which is serving previously in cluster but it is disconnected now
Following code snippet gives list of active brokers:
ZooKeeper zkInstance = new ZooKeeper("mymachine:port", 10000, null);
brokerIDs = zkInstance.getChildren("/brokers/ids", false);
for (String brokerID : brokerIDs) {
brokerInfo = new String(zkInstance.getData("/brokers/ids/" + brokerID, false, null));
String host=brokerInfo.substring(brokerInfo.indexOf("\"host\"")).split(",") [0].replaceAll("\"","").split(":")[1];
String port=brokerInfo.substring(brokerInfo.indexOf("\"jmx_port\"")).split(",") [0].replaceAll("\"","").split(":")[1];
System.out.println(host+":"+port);
}
Output:
my-machine-1:port
my-machine-2:port
my-machine-3:port
my-machine-4:port
I need information of all connected/disconnected brokers in multi node kafka cluster

Use describeCluster() of the AdminClient class to get the broker details such as host,port,id and rock.
Please refer the following code:
Properties kafkaProperties = new Properties();
kafkaProperties.put("bootstrap.servers", "localhost:9092,localhost:9093,localhost:9094");
AdminClient adminClient = AdminClient.create(kafkaProperties);
DescribeClusterResult describeClusterResult = adminClient.describeCluster();
Collection<Node> brokerDetails = describeClusterResult.nodes().get();
System.out.println("host and port details");
for(Node broker:brokerDetails) {
System.out.println(broker.host()+":"+broker.port());
}

Exception while trying to fetch data from zookeeper node: (KeeperErrorCode: ConnectionLoss )

I am using following code to extract Kafka broker list from zookeeper:
private static String getBrokerList() {
try {
ZooKeeper zookeeper = new ZooKeeper(zookeeperConnect, 15000, null);
List<String> ids = zookeeper.getChildren(ZkUtils.BrokerIdsPath(), false);
List<String> brokerList = new ArrayList<>();
for (String id : ids) {
String brokerInfo = new String(zookeeper.getData(ZkUtils.BrokerIdsPath() + '/' + id, false, null), Charset.forName("UTF-8"));
JsonObject jsonElement = new JsonParser().parse(brokerInfo).getAsJsonObject();
String host = jsonElement.get("host").getAsString();
brokerList.add(host + ':' + jsonElement.get("port").toString());
}
return Joiner.on(",").join(brokerList);
} catch (KeeperException | InterruptedException e) {
Throwables.propagate(e);
}
return "";
}
Above code is working fine when one thread executing the code at a time.
However, when several threads are executing the above code it fails with the following exception occasionally:
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/ids
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1560)
What am I doing wrong here?
My zookeeper version is 3.4.6-1569965.

from http://zookeeper.apache.org/doc/r3.4.9/api/org/apache/zookeeper/ZooKeeper.html#ZooKeeper(java.lang.String,%20int,%20org.apache.zookeeper.Watcher)
"Session establishment is asynchronous. This constructor will initiate connection to the server and return immediately - potentially (usually) before the session is fully established. The watcher argument specifies the watcher that will be notified of any changes in state. This notification can come at any point before or after the constructor call has returned."
You have to wait fro zookeeper connection to fully estabilish:
https://www.tutorialspoint.com/zookeeper/zookeeper_quick_guide.htm
Scroll down to the api section "Connect to the ZooKeeper Ensemble"

Retrieve multiple messages from SQS

I have multiple messages in SQS. The following code always returns only one, even if there are dozens visible (not in flight). setMaxNumberOfMessages I thought would allow multiple to be consumed at once .. have i misunderstood this?
CreateQueueRequest createQueueRequest = new CreateQueueRequest().withQueueName(queueName);
String queueUrl = sqs.createQueue(createQueueRequest).getQueueUrl();
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(queueUrl);
receiveMessageRequest.setMaxNumberOfMessages(10);
List<Message> messages = sqs.receiveMessage(receiveMessageRequest).getMessages();
for (Message message : messages) {
// i'm a message from SQS
}
I've also tried using withMaxNumberOfMessages without any such luck:
receiveMessageRequest.withMaxNumberOfMessages(10);
How do I know there are messages in the queue? More than 1?
Set<String> attrs = new HashSet<String>();
attrs.add("ApproximateNumberOfMessages");
CreateQueueRequest createQueueRequest = new CreateQueueRequest().withQueueName(queueName);
GetQueueAttributesRequest a = new GetQueueAttributesRequest().withQueueUrl(sqs.createQueue(createQueueRequest).getQueueUrl()).withAttributeNames(attrs);
Map<String,String> result = sqs.getQueueAttributes(a).getAttributes();
int num = Integer.parseInt(result.get("ApproximateNumberOfMessages"));
The above always is run prior and gives me an int that is >1
Thanks for your input

AWS API Reference Guide: Query/QueryReceiveMessage
Due to the distributed nature of the queue, a weighted random set of machines is sampled on a ReceiveMessage call. That means only the messages on the sampled machines are returned. If the number of messages in the queue is small (less than 1000), it is likely you will get fewer messages than you requested per ReceiveMessage call. If the number of messages in the queue is extremely small, you might not receive any messages in a particular ReceiveMessage response; in which case you should repeat the request.
and
MaxNumberOfMessages: Maximum number of messages to return. SQS never returns more messages than this value but might return fewer.

There is a comprehensive explanation for this (arguably rather idiosyncratic) behaviour in the SQS reference documentation.
SQS stores copies of messages on multiple servers and receive message requests are made to these servers with one of two possible strategies,
Short Polling : The default behaviour, only a subset of the servers (based on a weighted random distribution) are queried.
Long Polling : Enabled by setting the WaitTimeSeconds attribute to a non-zero value, all of the servers are queried.
In practice, for my limited tests, I always seem to get one message with short polling just as you did.

I had the same problem. What is your Receive Message Wait Time for your queue set to? When mine was at 0, it only returned 1 message even if there were 8 in the queue. When I increased the Receive Message Wait Time, then I got all of them. Seems kind of buggy to me.

I was just trying the same and with the help of these two attributes setMaxNumberOfMessages and setWaitTimeSeconds i was able to get 10 messages.
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(myQueueUrl);
receiveMessageRequest.setMaxNumberOfMessages(10);
receiveMessageRequest.setWaitTimeSeconds(20);
Snapshot of o/p:
Receiving messages from TestQueue.
Number of messages:10
Message
MessageId: 31a7c669-1f0c-4bf1-b18b-c7fa31f4e82d
...

receiveMessageRequest.withMaxNumberOfMessages(10);
Just to be clear, the more practical use of this would be to add to your constructor like this:
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(queueUrl).withMaxNumberOfMessages(10);
Otherwise, you might as well just do:
receiveMessageRequest.setMaxNumberOfMessages(10);
That being said, changing this won't help the original problem.

Thanks Caoilte!
I faced this issue also. Finally solved by using long polling follow the configuration here:
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-long-polling-for-queue.html
Unfortunately, to use long polling, you must create your queue as FIFO one. I tried standard queue with no luck.
And when receiving, need also set MaxNumberOfMessages. So my code is like:
ReceiveMessageRequest receive_request = new ReceiveMessageRequest()
.withQueueUrl(QUEUE_URL)
.withWaitTimeSeconds(20)
.withMaxNumberOfMessages(10);
Although solved, still feel too wired. AWS should definitely provide a more neat API for this kind of basic receiving operation.
From my point, AWS has many many cool features but not good APIs. Like those guys are rushing out all the time.

For small task list I use FIFO queue like stackoverflow.com/a/55149351/13678017
for example modified AWS tutorial
// Create a queue.
System.out.println("Creating a new Amazon SQS FIFO queue called " + "MyFifoQueue.fifo.\n");
final Map<String, String> attributes = new HashMap<>();
// A FIFO queue must have the FifoQueue attribute set to true.
attributes.put("FifoQueue", "true");
/*
* If the user doesn't provide a MessageDeduplicationId, generate a
* MessageDeduplicationId based on the content.
*/
attributes.put("ContentBasedDeduplication", "true");
// The FIFO queue name must end with the .fifo suffix.
final CreateQueueRequest createQueueRequest = new CreateQueueRequest("MyFifoQueue4.fifo")
.withAttributes(attributes);
final String myQueueUrl = sqs.createQueue(createQueueRequest).getQueueUrl();
// List all queues.
System.out.println("Listing all queues in your account.\n");
for (final String queueUrl : sqs.listQueues().getQueueUrls()) {
System.out.println(" QueueUrl: " + queueUrl);
}
System.out.println();
// Send a message.
System.out.println("Sending a message to MyQueue.\n");
for (int i = 0; i < 4; i++) {
var request = new SendMessageRequest()
.withQueueUrl(myQueueUrl)
.withMessageBody("message " + i)
.withMessageGroupId("userId1");
;
sqs.sendMessage(request);
}
for (int i = 0; i < 6; i++) {
var request = new SendMessageRequest()
.withQueueUrl(myQueueUrl)
.withMessageBody("message " + i)
.withMessageGroupId("userId2");
;
sqs.sendMessage(request);
}
// Receive messages.
System.out.println("Receiving messages from MyQueue.\n");
var receiveMessageRequest = new ReceiveMessageRequest(myQueueUrl);
receiveMessageRequest.setMaxNumberOfMessages(10);
receiveMessageRequest.setWaitTimeSeconds(20);
// what receive?
receiveMessageRequest.withMessageAttributeNames("userId2");
final List<Message> messages = sqs.receiveMessage(receiveMessageRequest).getMessages();
for (final Message message : messages) {
System.out.println("Message");
System.out.println(" MessageId: "
+ message.getMessageId());
System.out.println(" ReceiptHandle: "
+ message.getReceiptHandle());
System.out.println(" MD5OfBody: "
+ message.getMD5OfBody());
System.out.println(" Body: "
+ message.getBody());
for (final Entry<String, String> entry : message.getAttributes()
.entrySet()) {
System.out.println("Attribute");
System.out.println(" Name: " + entry
.getKey());
System.out.println(" Value: " + entry
.getValue());
}
}

Here's a workaround, you can call receiveMessageFromSQS method asynchronously.
bulkReceiveFromSQS (queueUrl, totalMessages, asyncLimit, batchSize, visibilityTimeout, waitTime, callback) {
batchSize = Math.min(batchSize, 10);
let self = this,
noOfIterations = Math.ceil(totalMessages / batchSize);
async.timesLimit(noOfIterations, asyncLimit, function(n, next) {
self.receiveMessageFromSQS(queueUrl, batchSize, visibilityTimeout, waitTime,
function(err, result) {
if (err) {
return next(err);
}
return next(null, _.get(result, 'Messages'));
});
}, function (err, listOfMessages) {
if (err) {
return callback(err);
}
listOfMessages = _.flatten(listOfMessages).filter(Boolean);
return callback(null, listOfMessages);
});
}
It will return you an array with a given number of messages