Develop Reference

Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected with Kafka Java client

I try to make Kafka producer Java codes on Windows 10. First, below is my Kafka producer codes.
Properties kafkaProdProperty = new Properties();
kafkaProdProperty.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, systemProp.get("kafka.brokerlist"));
kafkaProdProperty.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
kafkaProdProperty.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
kafkaProdProperty.put(ProducerConfig.ACKS_CONFIG, "0");
kafkaProdProperty.put(ProducerConfig.RETRIES_CONFIG, Integer.valueOf(1));
kafkaProdProperty.put(ProducerConfig.BATCH_SIZE_CONFIG, Integer.valueOf(20000));
kafkaProdProperty.put(ProducerConfig.LINGER_MS_CONFIG, Integer.valueOf(1));
kafkaProdProperty.put(ProducerConfig.BUFFER_MEMORY_CONFIG, Integer.valueOf(133554432));
Producer<String, String> kafkaProducer = new KafkaProducer<String, String>(kafkaProdProperty);
String line = //Read from hadoop file system continuously
ProducerRecord<String, String> kafkaProducerRecord = new ProducerRecord<String, String>(topic, line);
kafkaProducer.send(kafkaProducerRecord, new KafkaProducerCallback());
The execution of Kafka producer Java codes starts successfully without any exceptions. But the input line become longer and bigger, the warning message is thrown like below,
2021-07-27 10:30:28 WARN NetworkClient:780 - [Producer clientId=producer-8049] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2021-07-27 10:30:28 WARN NetworkClient:1077 - [Producer clientId=producer-8049] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected
2021-07-27 10:30:28 WARN NetworkClient:780 - [Producer clientId=producer-8049] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2021-07-27 10:30:28 WARN NetworkClient:1077 - [Producer clientId=producer-8049] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected
I think this issue is related with Kafka Java resources. But I have no idea what to configure. When the smaller input lines are set on Kafka producer parameter, it throws no warning messages like above.
How can I solve this issue?

unable to read kafka messages but can list the available topics

I am trying to connect locally from IntelliJ IDEA to kafka in k8s on minikube and read some messages. I can list the available topics with the consumer but am unable to read any messages.
The consumer code:
public class TestConsumer {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "consumer-test");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props, new StringDeserializer(), new StringDeserializer());
consumer.listTopics().forEach((key, value) -> System.out.println("topic = " + key));
consumer.subscribe(Collections.singletonList("test"));
try {
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1L));
for (ConsumerRecord<String, String> record : records)
System.out.println(record.offset() + ": " + record.value());
consumer.commitAsync();// doesn't work with or without this line
}
} finally {
consumer.close();
}
}
}
For installing in k8s I have used this helm chart https://github.com/bitnami/charts/tree/master/bitnami/kafka with the following configuration
replicaCount: 3
deleteTopicEnable: true
metrics:
kafka:
enabled: true
k8s services:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kafka ClusterIP 10.97.216.82 <none> 9092/TCP 30m
kafka-headless ClusterIP None <none> 9092/TCP,9093/TCP 30m
kafka-metrics ClusterIP 10.107.104.199 <none> 9308/TCP 30m
kafka-zookeeper ClusterIP 10.101.103.6 <none> 2181/TCP,2888/TCP,3888/TCP 30m
kafka-zookeeper-headless ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 30m
k8s pods:
NAME READY STATUS RESTARTS AGE
kafka-0 1/1 Running 4 32m
kafka-1 1/1 Running 4 32m
kafka-2 1/1 Running 2 32m
kafka-client 1/1 Running 0 31m
kafka-exporter-6ccc69f8cc-tgnxh 1/1 Running 0 33m
kafka-zookeeper-0 1/1 Running 0 33m
(only the kafka-client pod was manually created and not by the helm chart)
I am using this dependency of kafka compile group: 'org.apache.kafka', name: 'kafka-clients', version: '2.7.0'
As addition to the setup I run:
kubectl port-forward kafka-0 9092:9092
and add this line in my hosts file
127.0.0.1 kafka-headless.default.svc.cluster.local kafka-headless
Creation of the topic:
./kafka-topics.sh --create \
--zookeeper kafka-zookeeper:2181 \
--replication-factor 1 \
--partitions 1 \
--topic test
Producing on the topic
./kafka-console-producer.sh --broker-list kafka:9092 --topic test
I am able to read all the messages with the console consumer though
./kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test --from-beginning
Any help and suggestions are appreciated.

Thanks you guys! I finally managed to read messages from kafka.
#OneCricketeer points where the problem was and I found a similar link which explains the problem very well https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
What I did was to port forward kafka-0-external on port 9094 service and change the values for the helm chart to
replicaCount: 3
deleteTopicEnable: true
metrics:
kafka:
enabled: true
externalAccess:
enabled: true
service:
type: NodePort
port: 9094
autoDiscovery:
enabled: true
serviceAccount:
create: true
rbac:
create: true
extraEnvVars:
- name: KAFKA_LISTENERS
value: "CLIENT://:9092,INTERNAL://:9093,EXTERNAL://localhost:9094"
- name: KAFKA_ADVERTISED_LISTENERS
value: "CLIENT://:9092,INTERNAL://:9093,EXTERNAL://localhost:9094"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: "INTERNAL:PLAINTEXT,CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT"

Error while fetching metadata with correlation id 92 : {myTest=UNKNOWN_TOPIC_OR_PARTITION}

I have created a sample application to check my producer's code. My application runs fine when I'm sending data without a partitioning key. But, on specifying a key for data partitioning I'm getting the error:
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Error while fetching metadata with correlation id 37 : {myTest=UNKNOWN_TOPIC_OR_PARTITION}
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Error while fetching metadata with correlation id 38 : {myTest=UNKNOWN_TOPIC_OR_PARTITION}
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Error while fetching metadata with correlation id 39 : {myTest=UNKNOWN_TOPIC_OR_PARTITION}
for both consumer and producer. I have searched a lot on the internet, they have suggested to verify kafka.acl settings. I'm using kafka on HDInsight and I have no idea how to verify it and solve this issue.
My cluster has following configuration:
Head Node: 2
Worker Node:4
Zookeeper: 3
MY producer code:
public static void produce(String brokers, String topicName) throws IOException{
// Set properties used to configure the producer
Properties properties = new Properties();
// Set the brokers (bootstrap servers)
properties.setProperty("bootstrap.servers", brokers);
properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
// specify the protocol for Domain Joined clusters
//To create an Idempotent Producer
properties.setProperty(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
properties.setProperty(ProducerConfig.ACKS_CONFIG, "all");
properties.setProperty(ProducerConfig.RETRIES_CONFIG, Integer.toString(Integer.MAX_VALUE));
properties.setProperty(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "test-transactional-id");
KafkaProducer<String, String> producer = new KafkaProducer<>(properties);
producer.initTransactions();
// So we can generate random sentences
Random random = new Random();
String[] sentences = new String[] {
"the cow jumped over the moon",
"an apple a day keeps the doctor away",
"four score and seven years ago",
"snow white and the seven dwarfs",
"i am at two with nature",
};
for(String sentence: sentences){
// Send the sentence to the test topic
try
{
String key=sentence.substring(0,2);
producer.beginTransaction();
producer.send(new ProducerRecord<String, String>(topicName,key,sentence)).get();
}
catch (Exception ex)
{
System.out.print(ex.getMessage());
throw new IOException(ex.toString());
}
producer.commitTransaction();
}
}
Also, My topic consists of 3 partitions with replication factor=3

I made the replication factor less than the number of partitions and it worked for me. It sounds odd to me but yes, it started working after it.

The error clearly states that the topic (or partition) you are producing to does not exist.
Ultimately, you will need to describe the topic (via CLI kafka-topics --describe --topic <topicName> or other means) to verify if this is true
Kafka on HDInsight and I have no idea how to verify it and solve this issue.
ACLs are only setup if you installed the cluster with them, but I believe you can still list ACLs via zookeper-shell or SSHing into one of Hadoop masters.

I too had the same issue while creating a new topic. And when I described the topic, I could see that leaders were not assigned to the topic partitions.
Topic: xxxxxxxxx Partition: 0 Leader: none Replicas: 3,2,1 Isr:
Topic: xxxxxxxxx Partition: 1 Leader: none Replicas: 1,3,2 Isr:
After some googling, figured out that this could happen when we some issue with controller broker, so restarted the controller broker.
And Everything worked as expected...!

If the topic exists but you're still seeing this error, it could mean that the supplied list of brokers is incorrect. Check the bootstrap.servers value, it should be pointing to the right Kafka cluster where the topic resides.
I saw the same issue and I have multiple Kafka clusters and the topic clearly exists. However, my list of brokers was incorrect.

Kafka: This server is not the leader for that topic-partition

Possible duplicate of Kafka - This server is not the leader for that topic-partition but there is no accepted answer nor a clear solution.
I have a simple java program to produce the message to Kafka:
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("acks", "all");
props.put("retries", 1);
props.put("batch.size", 16384);
props.put("linger.ms", 100);
props.put("buffer.memory", 33554432);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "com.company.project.KafkaValueSerializer");
MyMessage message = new MyMessage();
Producer<String, MyMessage> producer = new KafkaProducer<>(props);
Future<RecordMetadata> future =
producer.send(new ProducerRecord<String, MyMessage>("My_Topic", "", message));
I am getting the following exception:
Exception in thread "main" java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.
at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:94)
at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:64)
at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29)
at
Caused by: org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.
When I tried with kafka-console-producer, I am getting the following error:
D:\kafka_2.11-0.10.2.0\bin\windows>kafka-console-producer.bat --broker-list localhost:9092 --topic My_Topic
hello
[2018-10-25 17:05:27,225] WARN Got error produce response with correlation id 3 on topic-partition My_Topic-0, retrying (2 attempts left). Error: NOT_LEADER_FOR_PARTITION (org.apache.kafka.clients.producer.internals.Sender)
[2018-10-25 17:05:27,335] WARN Got error produce response with correlation id 5 on topic-partition My_Topic-0, retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION (org.apache.kafka.clients.producer.internals.Sender)
[2018-10-25 17:05:27,444] WARN Got error produce response with correlation id 7 on topic-partition My_Topic-0, retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION (org.apache.kafka.clients.producer.internals.Sender)
[2018-10-25 17:05:27,544] ERROR Error when sending message to topic My_Topic with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.
When I describe my topic, I have the following information:
Topic:My_Topic PartitionCount:1 ReplicationFactor:1 Configs:
Topic: My_Topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
I tried creating a new topic and producing messages as mentioned in Quick start guide then the above steps work well.
What correction should I make in My_Topic or producer configuration so that I can publish messages successfully in this topic?

Had it been the case of "console client working, but Java program not working", then the solution of 'changing retry limit' may have helped.
As both the Java program and the built-in command-line producer are not able to connect to Kafka, I suspect that the problem could be due to stale configuration.
(Example: topic deleted and created again with the same name, but with different partition count).
Deleting the old log files of zookeeper and Kafka and creating the topic again, or creating a topic with a different name will resolve the issue.

Apache Kafka - Doesn't seem to load balance in Kafka Cluster

I am using Kafka 2.9.2-0.8.1 version..
From the documentation, it seems that load balancing is done automatically for the configured cluster.
Here is my Java producer configuration:
Properties props = new Properties();
props.put("batch.size", "200");
props.put("producer.type", "async");
props.put("connect.timeout.ms", "5000");
props.put("request.required.acks", "0");
props.put("metadata.broker.list", "10.10.73.52:9092,10.10.70.15:9092");
props.put("serializer.class", "kafka.serializer.DefaultEncoder");
props.put("partitioner.class", "kafka.producer.DefaultPartitioner");
Note: I have kept all the default configurations, which were provided by the Kafka distribution..
Zookeeper seems to have discovered my other broker : 10.10.70.15.. when I checked the logs..
I have created a test-topic on one of the brokers.. using the console-producer.sh.. which then created appropriate directory in /tmp/kafka-logs folder in all the other registered brokers in zookeeper.
--> ./kafka-topics.sh --create --zookeeper 10.10.73.52:2181 --replication-factor 2 --partitions 2 --topic test-topic
I have used the following provided line to subscribe to the topic on both broker machines..
--> ./kafka-console-consumer.sh --zookeeper 10.10.73.52:2181 --topic test-topic
Producer code:
KeyedMessage<String, byte[]> publishData = new KeyedMessage<String, byte[]>("test-topic", data);
producer.send(publishData);
I see that both the brokers receive the same data.. and load is not balanced.
Do I need to implement any other load-balance/partition logic?
Any ideas what am I missing here?