Develop Reference

Not getting avro messages while reading data from topic in java

I am writing code first time in java to consume AVRO data from kafka topic. I am using kafka-avro-console-producer to produce record.I am using leneseio/fast-data-dev image on Docker to UP kafka stack.
Producing record :
root#fast-data-dev / $ kafka-avro-console-producer --broker-list localhost:9092 --topic payengine --property schema.registry.url=http://localhost:8081 --property value.schema='{"type":"record", "name":"payengine", "fields":[{"name":"tin", "type":"string"},{"name":"ach","type":"string"}] }'
{"tin":"61582","ach":"I"}
{"tin":"97820","ach":"I"}
Now, to read this record, I have written below code. Also, it seems like I don't have to refer the schema while consuming records (as mentioned in the below reference link). I had also gone through one example, where SpecificAvroRecord was being used in place of GenericRecord, but that requires building class based on schema. I am not sure how GenericRecord points to correct schema, but don't see any schema reference in the example.
package com.github.psingh.Kafka;
import io.confluent.kafka.serializers.KafkaAvroDeserializer;
import io.confluent.kafka.serializers.KafkaAvroDeserializerConfig;
import org.apache.avro.generic.GenericRecord;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringSerializer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
public class SimpleConsumer_AvroSchema {
public static void main(String[] args) {
// System.out.println("Hello Kafka ");
// setting properties
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class.getName());
props.put(ConsumerConfig.GROUP_ID_CONFIG, "group1");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
props.put(KafkaAvroDeserializerConfig.SCHEMA_REGISTRY_URL_CONFIG, "http://localhost:8081");
//name topic
String topic = "payengine";
// create the consumer
KafkaConsumer<String, GenericRecord> consumer = new KafkaConsumer<String, GenericRecord>(props);
//subscribe to topic
consumer.subscribe(Collections.singleton(topic));
System.out.println("Waiting for the data...");
while (true) {
ConsumerRecords<String, GenericRecord> records = consumer.poll(Duration.ofMillis(5000));
for(ConsumerRecord<String,GenericRecord> record: records) {
System.out.print(record.value());
}
// consumer.commitSync();
}
}
}
The code built was successful.I was hoping to get console produced record here, but I am not getting anything :
Please suggest.
I have taken reference from here :
https://docs.confluent.io/current/schema-registry/serdes-develop/serdes-avro.html

Java Consumer for Kafka in Cloudera Quickstart not working

I have a cloudera Quickstart VM. i have installed Kafka parcels using Cloudera Manager and its working fine inside the VM using console based consumer and producer.
But when i try to use java based consumer it does not produce or consume messages. I can list the topics.
But i cannot consume messages.
following is my code.
package kafka_consumer;
import java.util.Arrays;
import java.util.Properties;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;
public class mclass {
public static void main(String[] args) {
Properties props = new Properties();
props.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "10.0.75.1:9092");
// Just a user-defined string to identify the consumer group
props.put(ConsumerConfig.GROUP_ID_CONFIG, "test");
// Enable auto offset commit
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");
props.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
// List of topics to subscribe to
consumer.subscribe(Arrays.asList("second_topic"));
for (String k_topic : consumer.listTopics().keySet()) {
System.out.println(k_topic);
}
while (true) {
try {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records) {
System.out.printf("Offset = %d\n", record.offset());
System.out.printf("Key = %s\n", record.key());
System.out.printf("Value = %s\n", record.value());
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
}
And following is the output of the code. while the console producer is producing messages but consumer is not able to receive it.
PS: i can telnet the port and ip of the Kafka broker. I can even list the topics. Consumer is constantly running without crashing but no messages is being consumed.

Lost message from the Kafka Topic

While trying timestamp in ProducerRecord; I found something weird. After sending few messages from the producer, I ran kafka-console-consumer.sh and verified that those messages are in the topic. I stopped the producer and waited for a minute. When I reran kafka-console-consumer.sh then it did not show the messages that I generated previously. I also added producer.flush() and producer.close() but the outcome was still the same.
Now, when I stopped using timestamp field then everything worked fine which makes me believe that there is something finicky about messages with timestamp.
I am using Kafka_2.11-2.0.0 (released on July 30, 2018)
Following is the sample code.
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.header.internal.RecordHeaders;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
import static java.lang.Thread.sleep;
public class KafkaProducerSample{
public static void main(String[] args){
String kafkaHost="sample:port";
String notificationTopic="test";
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, this.kafkaHost);
props.put(ProducerConfig.ACKS_CONFIG, 1);
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
Producer<String, String> producer = new KafkaProducer(props, new StringSerialize(), new StringSerializer);
RecordHeaders recordHeaders = new RecordHeader();
ProducerRecord<String, String> record = new ProducerRecord(notificationTopic, null, 1574443515L, sampleKey, SampleValue);
producer.send(record);
sleep(1000);
}
}
I run console consumer as following
$KAFKA_HOME/bin/kafka-console-consumer.sh --bootstrap.server KAFKA_HOST:PORT --topic test --from-beginning
#output after running producer
test
#output 5mins after shutting down producer

You are asynchronously sending only one record, but not ack-ing or flushing the buffer.
You will need to send more records,
or
producer.send(record).get();
or
producer.send(record);
producer.flush();
or (preferred), do Runtime.addShutdownHook() in your main method to flush and close the producer

kafka network Processor error in producer program(ArrayIndexOutOfBoundsException: 18)

I have below kafka producer Api program and i am new to kafka itself. Below code fetch data from one of API and send message to kafka topic.
package kafka_Demo;
import java.util.Properties;
import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import org.apache.kafka.clients.producer.*;
import java.net.URL;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
public class HttpBasicAuth {
public static void main(String[] args) {
try {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("zookeeper.connect", "localhost:2181");
props.put("batch.size", 16384);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer<>(props);
Json_read count = new Json_read();
URL url = new URL("https://alm.sysbiz.org/rest/api/2/search?jql=project=ALG&maxResults=0");
long total_ticket = count.ticketCount(url);
Alm_authentication alm = new Alm_authentication();
for (long i = 0; i <=total_ticket ; i = i + 20) {
url = new URL("https://alm.sysbiz.org/rest/api/2/search?jql=project=ALG&expand=changelog&startAt=" + i+"&maxResults=20");
InputStream content = (InputStream) alm.performAuth(url);
if (content != null) {
BufferedReader in = new BufferedReader(new InputStreamReader(content));
String line;
while ((line = in.readLine()) != null) {
//writer.println(line);
// System.out.println(line);
producer.send(new ProducerRecord<String, String>("test", "a11", line)).get();
}
}
producer.send(new ProducerRecord<String, String>("test", "a11", "\n"));
content.close();
}
producer.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
While i am running above code i am getting below error at kafka broker.
ERROR Processor got uncaught exception. (kafka.network.Processor)
java.lang.ArrayIndexOutOfBoundsException: 18
at org.apache.kafka.common.protocol.ApiKeys.forId(ApiKeys.java:68)
at org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:39)
at kafka.network.RequestChannel$Request.<init>(RequestChannel.scala:79)
at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:426)
at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:421)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at kafka.network.Processor.run(SocketServer.scala:421)
at java.lang.Thread.run(Thread.java:748)
I have tried a lot by searching in Google but no luck. Can any one help me to resolve it.
And below error i can see it in my eclipse IDE.
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1124)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:823)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:760)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:648)
at kafka_Demo.HttpBasicAuth.main(HttpBasicAuth.java:40)
Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

This happens due to kafka version mismatch.
Make sure kafka-clients jar version & kafka server version installed in your machine.

I have faced similar issue even my eclipse and kafka is having same jar , Then i have found the root cause : i have installed another version of zookeeper in my machine for solr configuration and that zookeeper lib was added in classpath of environment variable. so starting kafka-zookeeper it was referring to another lib so i deleted the classpath for zookeeper and tried with kafka-zookeeper ,IT Worked :)

kafka produce and send message from java app and consume in cli

I am new to kafka. I am trying to send message through java app and consume it in command line prompt, but the message is not getting displayed on CLI.
Following is the java code:
package com.kafka.prj;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.util.Properties;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.KafkaProducer;
public class KafkaProd {
private static KafkaProducer<String, String> producer;
public void initialize() {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 16384);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
// ProducerConfig producerConfig = new ProducerConfig(producerProps);
producer = new KafkaProducer<String, String>(props);
}
public void publishMesssage() throws Exception{
producer.send(new ProducerRecord<String, String>("test1", "dummy text msg"));
return;
}
public static void main(String[] args) {
KafkaProd kafkaProducer = new KafkaProd();
// Initialize producer
kafkaProducer.initialize();
// Publish message
try {
kafkaProducer.publishMesssage();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
//Close the producer
producer.close();
}
}
In CLI, following is the command am using to consume the message sent in code above:
$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test1 --from-beginning
The above command displays nothing, no error, no output.
Where am I getting wrong?

I was having the same problem, ie messages produced by kafka-console-producer.sh were visible on the kafka-console-consumer.sh console. But, on using the Java producer, kafka-console-consumer.sh console did not receive any messages. Also, the logs on the Java producer had this as the last line :
2017-07-10 16:36:42 INFO KafkaProducer:972 - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
This means that the Java producer is not able to connect to the Kafka brokers as given by the bootstrap.servers config on the Java producer. (Although, I was actually able to telnet to the broker port from the same machine as the Java producer ). Solution is to add the property : advertised.host.name on all the brokers. and make it equal to the IP/hostname of the broker. This should be in line with whatever you are providing in the bootstrap.servers.
My bootstrap.servers had the value - 192.168.10.12:9092,192.168.10.13:9092,192.168.10.14:9092, so on each of the brokers, I made advertised.host.name=192.168.10.12, advertised.host.name=192.168.10.13 and advertised.host.name=192.168.10.14 respectively.