Develop Reference

Why do we need to use threads for running Kafka consumers? And how many threads do we need?

I am relatively new to Java (I have some experience with Scala) and currently trying to learn about Kafka. I came across the following example in this tutorial (I am adding the code mostly for reference):
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.errors.WakeupException;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.time.Duration;
import java.util.Arrays;
import java.util.Collections;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
public class ConsumerDemoWithThread {
public static void main(String[] args) {
new ConsumerDemoWithThread().run();
}
private ConsumerDemoWithThread() {
}
private void run() {
Logger logger = LoggerFactory.getLogger(ConsumerDemoWithThread.class.getName());
String bootstrapServers = "127.0.0.1:9092";
String groupId = "my-sixth-application";
String topic = "first_topic";
// latch for dealing with multiple threads
CountDownLatch latch = new CountDownLatch(1);
// create the consumer runnable
logger.info("Creating the consumer thread");
Runnable myConsumerRunnable = new ConsumerRunnable(
bootstrapServers,
groupId,
topic,
latch
);
// start the thread
Thread myThread = new Thread(myConsumerRunnable);
myThread.start();
// add a shutdown hook
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
logger.info("Caught shutdown hook");
((ConsumerRunnable) myConsumerRunnable).shutdown();
try {
latch.await();
} catch (InterruptedException e) {
e.printStackTrace();
}
logger.info("Application has exited");
}
));
try {
latch.await();
} catch (InterruptedException e) {
logger.error("Application got interrupted", e);
} finally {
logger.info("Application is closing");
}
}
public class ConsumerRunnable implements Runnable {
private final CountDownLatch latch;
private final KafkaConsumer<String, String> consumer;
private final Logger logger = LoggerFactory.getLogger(ConsumerRunnable.class.getName());
public ConsumerRunnable(String bootstrapServers,
String groupId,
String topic,
CountDownLatch latch) {
this.latch = latch;
// create consumer configs
Properties properties = new Properties();
properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, groupId);
properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
// create consumer
consumer = new KafkaConsumer<String, String>(properties);
// subscribe consumer to our topic(s)
consumer.subscribe(Collections.singletonList(topic));
}
#Override
public void run() {
// poll for new data
try {
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofMillis(100)); // new in Kafka 2.0.0
for (ConsumerRecord<String, String> record : records) {
logger.info("Key: " + record.key() + ", Value: " + record.value());
logger.info("Partition: " + record.partition() + ", Offset:" + record.offset());
}
}
} catch (WakeupException e) {
logger.info("Received shutdown signal!");
} finally {
consumer.close();
// tell our main code we're done with the consumer
latch.countDown();
}
}
public void shutdown() {
// the wakeup() method is a special method to interrupt consumer.poll()
// it will throw the exception WakeUpException
consumer.wakeup();
}
}
}
I am mainly trying to understand:
What are the benefits of using a thread for running the
consumer? I (I thought Kafka abstracts the distribution of load across consumers anyways)
When we use Thread myThread = new Thread(myConsumerRunnable); does
that run in a single thread or across multiple threads?
Why do we fire the shutdown hook via a separate thread? (To my understanding from inspecting the method, it seems more like a Java thing than a Kafka thing)

What are the benefits of using a thread for running the consumer? I (I
thought Kafka abstracts the distribution of load across consumers
anyways)
As you can see, the consumer starts an infinite loop in the run method. Starting it as a new thread lets you do more things in the main thread while the consumer is already active.
When we use Thread myThread = new Thread(myConsumerRunnable); does that run in a > single thread or across multiple threads?
Creating the Thread object does not start the new thread yet. It's myThread.start(); where the new thread starts execution. Your example program has a main thread and a consumer thread. The main thread actually waits for the shutdown signal through the CountDownLatch latch, so arguably it would have been possible to avoid the consumer thread.
Why do we fire the shutdown hook via a separate thread? (To my understanding
from inspecting the method, it seems more like a Java thing than a Kafka thing)
It's a java thing. The shutdown hook thread is not actually executed until shutdown happens. See https://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html#addShutdownHook(java.lang.Thread)

worker thread is not notified in multithread code

I want to write a traditional producer and consumer multithread program in Java. Producer thread will send message to thread-safe List until this list get full. once buffer is full, it will notify worker thread and buffer will be cleared. In my coding, the worker thread is not notified.
Do you know the reason ? Thanks.
package com;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.locks.Condition;
import java.util.concurrent.locks.ReentrantLock;
public class ThreadTest {
public static void main(String[] args) {
final List<String> bufferSafeList = Collections.synchronizedList(new ArrayList<String>());
final ReentrantLock bufferLock = new ReentrantLock();
final Condition bufferFull = bufferLock.newCondition();
// final Condition bufferEmpty = bufferLock.newCondition();
Thread producerThread = new Thread(new Runnable() {
#Override
public void run() {
while (true) {
try {
bufferLock.lock();
bufferSafeList.add(System.currentTimeMillis() + "");
System.out.println("add to buffer " + bufferSafeList.size());
if (bufferSafeList.size() > 100) {
System.out.println("send wake up signal");
bufferFull.signalAll();
//waiting for buff cleared
while(!bufferSafeList.isEmpty()){
Thread.sleep(1000);
}
}
Thread.sleep(1000);
} catch(Exception e){
e.printStackTrace();
}
finally {
bufferLock.unlock();
}
}
}
});
producerThread.start();
Thread workerThread = new Thread(new Runnable() {
#Override
public void run() {
while (true) {
try {
bufferLock.lock();
System.out.println("waiting for wakeup signal");
bufferFull.await();
System.out.println("clear buffer");
bufferSafeList.clear();
} catch (Exception ex) {
ex.printStackTrace();
} finally {
bufferLock.unlock();
}
}
}
});
workerThread.start();
}
}

You only unlock the bufferLock in the finally, so the workerThread will never be able to progress further than trying to obtain the lock
When the buffer is full unlock so that the workerThread can continue

If the producer thread happens to run first, as it is likely to do because it is started first, it will likely lock bufferLock first, in which case it will fill the list and invoke bufferFull.signalAll() while the consumer is still waiting to acquire the lock. You then busy-wait for the worker to clear the list, which it never will do because it cannot proceed.
Moreover, even if you unlocked the lock after signalling, that would be too late. By the time the worker is await()ing the condition, the signal has already come and gone. Indeed, there is a race condition there even if the producer waits to acquire the lock until it is ready to signal.
The correct usage of a condition variable always involves testing whether the expected condition is satisfied before waiting, and looping back to wait some more if it is not satisfied after resuming from the wait. You can overcome the race in your code by implementing this.

Java ExecutorService - why does this program keep running?

I'm trying to build something like a background task executor which terminates background tasks after a certain time if there's no answer (background tasks call webservices and they can time-out but I need to make sure they time out under a certain time)
So I have this as an experiment but if I run this the program does not terminate. I wonder if its because a background thread is still active maybe? How can I shut this down?
public class Test {
public static class Task implements Callable<Object> {
#Override
public Object call() throws Exception {
while(true) {}
}
}
public static void main(String[] args) {
try {
Task t = new Task();
ExecutorService executor = Executors.newSingleThreadExecutor();
executor.invokeAll(Arrays.asList(t), 5L, TimeUnit.SECONDS);
executor.shutdown();
System.out.println("DONE");
}
catch (InterruptedException e) {
e.printStackTrace();
}
}
}

The ExecutorService does not kill the running threads, and since threads are created as non-daemon, the JVM doesn't exit.
What happens is that when timeout expires, futures returned by invokeAll() are cancelled, which means that a flag is set on the future object and you get a CancellationException if you try to call future.get(). However neither invokeAll(), nor shutdown() (or shutdownNow()) do anything to kill the thread.
Note that you cannot even kill threads yourself. All you can do is setting some application-specific flag or call Thread.interrupt(), but even that does not guarantee that the thread terminates.

There is a great post by Winterbe on how executors work. This is an excerpt from his tutorial
So basically executors always keep listening to the new tasks or callables/runnables and one way to shutdown the executor or stop the executor from listening is to interrupt whatever task it is executing. One way to do is calling the future.get() which stops when the main thread , suspends it and makes sure that the current thread gets executed completely before handing over the resource to other thread
You could probably have a higher number of threads and write your code to shutdown gracefully in the InterruptedException block
Here is a sample code that I've written and tested:
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
public class ExecutorTest {
public static void main(String[] args) {
ExecutorService service = Executors.newWorkStealingPool(10);
Callable<AccountClass> newInstance = () -> {
TimeUnit.SECONDS.sleep(3);
return getAcc(Thread.currentThread().getId());
};
// for now only one instance is added to the list
// List<Callable<AccountClass>> callablesSingleList = Arrays.asList(newInstance);
// adding multipleCallalbes
List<Callable<AccountClass>> callablesMultipleList = Arrays.asList(
() -> {
TimeUnit.SECONDS.sleep(3);
return getAcc(Thread.currentThread().getId());
},
() -> {
TimeUnit.SECONDS.sleep(3);
return getAcc(Thread.currentThread().getId());
},
() -> {
TimeUnit.SECONDS.sleep(3);
return getAcc(Thread.currentThread().getId());
});
try {
service.invokeAll(callablesMultipleList).stream().map(future -> {
AccountClass fuClass = null;
try {
fuClass = future.get();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
return fuClass;
}).forEach(getValue -> {
System.out.println("retunred value:" + getValue);
});
} catch (InterruptedException ex) {
ex.printStackTrace();
}
}
private static AccountClass getAcc(long itr) {
// probably call DB for every new thread iterator
System.out.println("getting the current thread" + itr);
AccountClass ac = new AccountClass();
ac.setId(itr);
ac.setName("vv");
ac.setRole("admin");
System.out.println("sending the accnt class:" + ac);
return ac;
}
}
UPDATE:
Another way of shutting down the executor is using the service.shutDownNow() - > which shutdowns the program even if its the middle of execution. You could use awaitTermination method to specify if you feel that it might take a few minutes to complete execution and then probably shutdown the service
import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
public class ExecutorScheduleFixedRate {
public static void main(String[] args) {
ScheduledExecutorService service = Executors.newScheduledThreadPool(10);
Runnable task = () -> {
getAcc(33);
};
service.scheduleWithFixedDelay(task, 10, 5, TimeUnit.SECONDS);
if (!service.isShutdown()) {
List<Runnable> list2 = service.shutdownNow();
System.out.println(list2);
System.out.println("is shutdonw" + service.isShutdown());
System.out.println("Do something after the thread execution");
}
}
private static AccountClass getAcc(long itr) {
// probably call DB for every new thread iterator
System.out.println("getting the current thread" + itr);
AccountClass ac = new AccountClass();
ac.setId(itr);
ac.setName("vv");
ac.setRole("admin");
System.out.println("sending the accnt class:" + ac);
return ac;
}
}

Producer consumer using blockingqueue

I have started learning threads and tried Producer consumer problem in Java using concurrent package introduced in JDK 5.0 I have written the following code:
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
class Producer implements Runnable {
private final BlockingQueue<Integer> objqueue;
Producer(BlockingQueue<Integer> obj) {
objqueue = obj;
}
#Override
public void run() {
int i = 0;
while (i < 10) {
try {
System.out.println("Put : " + i);
objqueue.put(i);
} catch (InterruptedException e) {
}
i++;
}
}
}
class Consumer implements Runnable {
private final BlockingQueue<Integer> objqueue;
Consumer(BlockingQueue<Integer> obj) {
objqueue = obj;
}
#Override
public void run() {
while (true) {
try {
System.out.println("Got : " + objqueue.take());
} catch (InterruptedException e) {
}
}
}
}
public class PCMain {
public static void main(String[] args) {
// create shared object
BlockingQueue<Integer> obj = new LinkedBlockingQueue<Integer>();
Thread prod = new Thread(new Producer(obj));
Thread cons = new Thread(new Consumer(obj));
prod.start();
cons.start();
}
}
The program is not terminating when the producer has produced up to 9 and consumer consumed up to 9. Should I remove the while loop which is true forever in Consumer.
How can I make it for more than one Producer and one Consumer?
Thanks.

Well you have two threads, one should stop once i == 10. The other thread is in an infinite loop though. You need to signal to the consuming thread that the application should end. Look at the Poison Pill as a way of telling the second thread to stop.
The program itself won't stop until that consuming thread completed.

Removing while loop will cause consumer will consume only 1 object given by producer.
Better to go Excecuter framework. It is having Thread Factory and Thread Pool.You can use to implement the same.

I think the easiest way to "fix" your code is to make the consumer a daemon thread.
Thread prod = new Thread(new Producer(obj));
Thread cons = new Thread(new Consumer(obj));
cons.setDaemon( true );
prod.start();
cons.start();
This really isn't a general solution, but a good trick to keep in mind when it's inconvenient to signal a thread to stop.

LinkedBlockingQueue failing to wait for the threads to execute

I am making a WordCounter, which has several threads counting the words in different files. I have gotten everything to work, except one little issue.
I cannot figure out a proper way to wait for the threads to finish. Everything works if I set a Thread.sleep to wait for a short amount of time, the only problem is that, this will not be a proper solution if the counter takes longer than the sleep does.
import java.io.*;
import java.util.*;
import java.util.concurrent.BlockingQueue;
public class WordCounter implements Runnable{
private String file;
private BlockingQueue<Integer> q;
private int words = 0;
public WordCounter(String f, BlockingQueue<Integer> queue){
file = f;
q = queue;
}
public void run(){
try{
Scanner in = new Scanner(new File(file));
while (in.hasNext()){
in.next();
words++;
}
in.close();
System.out.println(file + ": " + words);
q.add(words);
}
catch (FileNotFoundException e){
System.out.println(file + " blev ikke fundet.");
}
}
}
This is the code from the actual word-counter. I want my main-thread to wait for these word-counter threads to do the q.add(words); function before doing anything else.
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
public class MainThread implements Runnable{
private String[] arguments;
public MainThread(String[] args){
arguments = args;
}
public void run() {
final BlockingQueue<Integer> queue = new LinkedBlockingQueue<Integer>();
for(String f : arguments){
WordCounter c = new WordCounter(f, queue);
Thread t = new Thread(c);
t.start();
}
while(!queue.isEmpty()){
try {
System.out.println(queue.take());
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
This is the main-thread. I will need some sort of way to wait for the other threads to finish before I continue to my while statement at the bottom, but how?
Thanks in advance.

Use an ExecutorService and wait on the Future returned. The code below will submit each task to a thread within the executor service (thread pool) and get back the future for that task. When all submitted it will wait on the future. The get method will only return when the run method completes in the task.
public class MainThread implements Runnable{
private String[] arguments;
public MainThread(String[] args){
arguments = args;
}
public void run() {
ExecutorService e = Executors.newFixedThreadPool(arguments.length);
final BlockingQueue<Integer> queue = new LinkedBlockingQueue<Integer>();
List<Future<?>> tasks = new ArrayList<>();
for(String f : arguments){
tasks.add(e.submit(new WordCounter(f, queue)));
}
for(Future<?> f : tasks) {
f.get();
}
while(!queue.isEmpty()){
try {
System.out.println(queue.take());
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
However
You can make your code cleaner by removing the BlockingQueue entirely and having each task be a Callable<Integer> where it simply returns the words variable. And when you call future.get() the return value there would be the word count.

This is what i would do:
create a counter variable (here is how to do it in a way that is safe for multi-threads) to keep track of how many threads you are spawning in the main
thread
create an interface with function signatures to
increment/decrement that counter
implement that interface in your
main thread
subclass a worker thread to accept that interface as
a parameter
once the worker thread finishes, call that interface
to decrement the number of running threads.
in the implementation of the decrement function on the main thread, add a
condition to do something once the counter is 0.

If you know how many threads to wait on, then you can use a shared semaphore. The worker threads each calls release on the semaphore when they're done; the main thread calls acquire(n) where n is the number of worker threads, which causes the main thread to wait until n permits are available (i.e. until all n worker threads are finished).

You need to keep the created threads in a list and join them from the current thread.
Something like this:
List<Thread> threads = new LinkedList<Thread>();
for (String f : arguments) {
WordCounter c = new WordCounter(f, queue);
Thread t = new Thread(c);
t.start();
threads.add(t);
}
for (Thread t : threads) {
t.join();
}
The join() method will block until the thread terminates.