Develop Reference

Semaphore in Java. producer-consumer problem

I am testing the use of semaphores with the typical producer-consumer problem where I only have one producer and one consumer. The producer adds products one at a time and the consumer can withdraw several simultaneously.
To perform the test, the producer and the consumer store and remove numbers from a array of 10 elements where 0 represents that there are no products and any other number represents a product. Access to store and retrieve items is centralized in a class called Data. I use a mutex to make an orderly use of the vector in case we have more than one thread working simultaneously.
When executing it, I observe that the number of permissions is not correct according to the operations performed by the threads. The application shows an error because the semaphore of the producer says that it has permission, but the data vector is full.
package producer.consumer;
import java.io.IOException;
public class ProducerConsumer {
public static void main(String[] args) throws IOException {
final int MAX = 10;
Data data = new Data(MAX);
Consumer consumer = new Consumer(data);
Producer producer = new Producer(data);
consumer.start();
producer.start();
}
}
package producer.consumer;
public class Producer extends Thread{
private final Data data;
public Producer(Data data) {
this.data = data;
}
#Override
public void run() {
while (true) {
try {
data.add((int) (Math.random() * data.getLength()) + 1);
} catch (InterruptedException ex) {
System.out.println(ex.getMessage());
}
}
}
}
package producer.consumer;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Consumer extends Thread{
private final Data data;
public Consumer(Data data) {
this.data = data;
}
#Override
public void run() {
while (true) {
try {
data.remove((int) (Math.random() * data.getLength()) + 1);
} catch (InterruptedException ex) {
Logger.getLogger(Consumer.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
}
package producer.consumer;
import java.io.IOException;
import java.util.Arrays;
import java.util.concurrent.Semaphore;
public class Data {
private final int[] data;
private final Semaphore mutex = new Semaphore(1);
private final Semaphore semProducer, semConsumer;
public Data(int MAX) throws IOException {
data = new int[MAX];
semProducer = new Semaphore(MAX);
semConsumer = new Semaphore(0);
}
public int getLength() {
return data.length;
}
public void add(int number) throws InterruptedException {
semProducer.acquire();
mutex.acquire();
System.out.println("trying to add a product");
int i = 0;
while (data[i] != 0) {
i++;
}
data[i] = number;
int permits = semConsumer.availablePermits() + 1;
System.out.println("data added in " + i + " " + Arrays.toString(data)
+ " Resources consumer " + permits
+ " Resources producer " + semProducer.availablePermits());
mutex.release();
semConsumer.release();
}
public void remove(int numberElements) throws InterruptedException {
semConsumer.acquire(numberElements);
mutex.acquire();
System.out.println("trying to withdraw " + numberElements);
for (int i = 0; i < numberElements; i++) {
if (data[i] != 0) {
data[i] = 0;
}
}
int permisos = semProducer.availablePermits() + 1;
System.out.println(" Retired " + numberElements + " " + Arrays.toString(data)
+ " Resources consumer " + semConsumer.availablePermits()
+ " Resources producer " + permisos);
mutex.release();
semProducer.release(numberElements);
}
}
Thank you very much for the help.

Your consumer does not always consume what it claims to consume.
for (int i = 0; i < numberElements; i++) {
if (data[i] != 0) {
data[i] = 0;
}
}
Suppose numberElements is 3, and that we have exactly 3 available elements in data[7], data[8], data[9].
The loop terminates with i == 3, nothing has been removed, but the producer semaphore will still be 'upped' by 3.
In the consumer, if you use i as the array index, it needs to cover the whole array, and you need a separate counter for 'elements removed'.
It is not the case that available elements will always be in the lowest-numbered data slots even though the producer fills those in first. Consider the time sequence that the producer manages to produce at least 5 elements, then the consumer runs to consume 2, and then immediately runs again to consume 3, before any more have been produced. data[0] and data[1] will be empty on the second run of the consumer and we run into the scenario I describe.

EDIT Acquiring and releasing permits seems correct; but you need to make sure that the consumer will actually clear the correct number of elements.
In example, edit the Data class with
public void remove(int numberElements) throws InterruptedException {
semConsumer.acquire(numberElements);
mutex.acquire();
System.out.println("remove: num-elem=" + numberElements);
int consumed=0;
for (int i = 0; consumed<numberElements; i++) {
if (data[i] != 0) {
data[i] = 0;
consumed++;
}
}
System.out.println(
" Retired " + numberElements + " " + Arrays.toString(data) );
mutex.release();
semProducer.release(numberElements);
}
Note also that this implementation is not very efficient (you'll need to iterate over the whole array both when inserting and deleting items, which can be expensive when MAX is large..)

Java multi-threading example printing message 100 times on different threads?

I'm tracing this code and I'm trying to figure out what exactly it's supposed to do. I can't get it running on IntelliJ. The run option is greyed out even though I defined the Project SDK. But I just want to know what the code is supposed to do.
I just read a bit of theory on threads. Is it just supposed to display each message 100 times with a timestamp on different threads? And Runnable 4 is an example of how to do it with a lambda correct?
Main class
import java.util.Date;
import java.util.concurrent.*;
public class Example02
{
public static void main(String []args)
{
// create runnables
PrintMessageRunnable pmRunnable1 = new PrintMessageRunnable("Runnable 1");
PrintMessageRunnable pmRunnable2 = new PrintMessageRunnable("Runnable 2");
PrintMessageRunnable pmRunnable3 = new PrintMessageRunnable("Runnable 3");
// passing a runnable using Lambda notation
Runnable pmRunnable4 = () -> {
// this is the code inside the run method
String message = "Lambda Runnable";
int REPETITIONS = 100;
int DELAY = 100;
try {
for(int i = 1; i <= REPETITIONS; i++) {
Date now = new Date();
System.out.println(now + ": " + message + "." + i);
Thread.sleep(DELAY);
}
}
catch (InterruptedException e) {
System.out.println("Runnable version interrupted.");
}
};
// specify how many threads the executor service should manage
int MAX_THREADS = 2;
ExecutorService pool = Executors.newFixedThreadPool(MAX_THREADS);
// start running
pool.execute(pmRunnable1);
pool.execute(pmRunnable2);
pool.execute(pmRunnable3);
pool.execute(pmRunnable4);
}
}
Print Message Runnable class
import java.util.*;
public class PrintMessageRunnable implements Runnable
{
private String message;
private int REPETITIONS = 100;
private int DELAY = 100;
public PrintMessageRunnable(String message){
this.message = message;
}
public void run(){
try {
for(int i = 1; i <= REPETITIONS; i++) {
Date now = new Date();
System.out.println(now + ": " + message + "." + i);
Thread.sleep(DELAY);
}
}
catch (InterruptedException e) {
System.out.println("Runnable version interrupted.");
}
}
}

In your example you have 2 threads which prints your message with a timestamp.
The lambda presentation of runnable is correct too.
But the usage of java.util.Date is dangerous, bacause of it isn't threadsafe.
Use LocalDateTime in multithread application to avoid errors

Why is this program creating more threads than possible?

This is for a custom UDTF in a hive query, CreateLogTable is the UDTF class which I am using as a temp for testing. I am creating one thread per file to be downloaded from Amazon S3 and waiting until another thread becomes available before allocating another file to the thread.
Main Test logic:
CreateLogTable CLT = new CreateLogTable();
int numThreads = 2;
int index = 0;
DownloadFileThread[] dlThreads = new DownloadFileThread[numThreads];
for (S3ObjectSummary oSummary : bucketKeys.getObjectSummaries()) {
while (dlThreads[index] != null && dlThreads[index].isAlive()) {
index += 1;
index = index % numThreads;
}
dlThreads[index] = new DownloadFileThread(CLT , getBucket(oSummary.getBucketName() + "/"
+ oSummary.getKey()), getFile(oSummary.getKey()), index);
dlThreads[index].start();
index += 1;
index = index % numThreads;
}
Thread class (run() method):
try {
System.out.println("Creating thread " + this.threadnum);
this.fileObj = this.S3CLIENT.getObject(new GetObjectRequest(this.filePath, this.fileName));
this.fileIn = new Scanner(new GZIPInputStream(this.fileObj.getObjectContent()));
while (this.fileIn.hasNext()) {
this.parent.forwardToTable(fileIn.nextLine());
}
System.out.println("Finished " + this.threadnum);
} catch (Throwable e) {
System.out.println("Downloading of " + this.fileName + " failed.");
}
The while loop before the thread creation should be looping until it finds a null thread or a dead thread until it exits the loop, in which case a new thread will be created and started. Since I included logging to console, I am able to observe this process, but the output is unexpected:
Creating thread 0
Creating thread 1
Creating thread 0
Creating thread 1
Creating thread 0
Creating thread 1
Creating thread 0
...
Creating thread 1
Creating thread 0
Creating thread 1
Finished 0
Finished 1
Finished 1
Finished 0
Finished 1
Finished 1
...
Finished 0
Finished 1
Finished 0
Finished 1
The above is only the first few lines of output. The issue is that more than two threads are created before any threads complete their tasks.
Why is this happening and how can I fix this?

I reduced your code to this test case:
public class ThreadTest {
private static class SleepThread extends Thread {
private final int index;
SleepThread(int ii) { index = ii; }
#Override
public void run() {
System.out.println("Creating thread " + this.index);
try {
Thread.sleep(5_000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Finished " + this.index);
}
}
public static void main(String[] args) {
int numThreads = 2;
int index = 0;
SleepThread[] dlThreads = new SleepThread[numThreads];
for (int ii = 0; ii < 10; ++ii) {
while (dlThreads[index] != null && dlThreads[index].isAlive()) {
index += 1;
index = index % numThreads;
}
dlThreads[index] = new SleepThread(index);
dlThreads[index].start();
index += 1;
index = index % numThreads;
}
}
}
Using Sun JDK 1.7.0_75, running this produces the result that you'd expect--two threads start, they exit after five seconds, two more threads start, and so on.
The next thing I'd suspect is that your JVM's implementation of Thread.isAlive() isn't returning true for threads immediately after they are started, although that seems contrary to the documentation for the Thread class.

Try to see this example:
public static void main(String[] args) {
ExecutorService executor = Executors.newFixedThreadPool(5);
for (int i = 0; i < 10; i++) {
Runnable worker = new WorkerThread("" + i);
executor.execute(worker);
}
executor.shutdown();
while (!executor.isTerminated()) {
}
System.out.println("Finished all threads");
}
It's a thread pool using Java 8. A very simple and esay way to make it using the Executors. Very staraight forward way to make it.

The reason why the above code wasn't working was because of something wacky going on with the call to isAlive().
For some reason, no matter what state a thread is in, isAlive() will always return false for me, causing the creation of more and more threads, which replace the old ones in the array, dlThreads.
I solved the issue by creating a custom isWorking() method which simply returns a boolean of whether or not the thread's run() method has completed. Here is what the Thread class looks like now:
//this.isWorking initialized to true during instantiation
#Override
public void run() {
try {
System.out.println("Creating thread " + this.threadnum + " for " + filePath + "/" + fileName);
this.fileObj = this.S3CLIENT.getObject(new GetObjectRequest(this.filePath, this.fileName));
this.fileIn = new Scanner(new GZIPInputStream(this.fileObj.getObjectContent()));
while (this.fileIn.hasNext()) {
this.parent.forwardToTable(fileIn.nextLine());
}
System.out.println("Finished " + this.threadnum);
this.isWorking = false;
} catch (Throwable e) {
System.out.println("Downloading of " + this.fileName + " failed.");
e.printStackTrace();
this.isWorking = false;
}
}
public boolean isWorking(){
return this.isWorking;
}
However, after implementing this and being satisfied that my multithreaded script works, I switched over to using an Executor, as suggested by other users, which slightly improved performance and made the code much cleaner.

How to create a cyclic exchange of three threads?

How to create a cyclic exchange of three threads? That is: first thread must send data to second, second to third and third thread must send data to first.
I wrote some code, but threads exchange in random oder.
class DataClass {
int value;
String message;
DataClass(int v, String s) {
value = v;
message = s;
}
int getValue() {
return (value);
}
String getMassage() {
return (message);
}
}
class Loop implements Runnable {
int counter;
String name;
Exchanger<DataClass> exchanger;
Loop(int startValue, String id, Exchanger<DataClass> ex) {
counter = startValue;
name = id;
exchanger = ex;
System.out.println(name + ": created");
}
public void run() {
System.out.println(name + ": started");
DataClass data = new DataClass(counter, name);
for (int i = 0; i < 3; ++i) {
try {
DataClass newData = exchanger.exchange(data);
counter += newData.getValue();
System.out.println(name + ": from "
+ newData.getMassage() + ": data: "
+ newData.getValue() + ": state = " + counter);
} catch (InterruptedException e) {
System.err.println(e.toString());
}
}
System.out.println(name + ": ended");
}
}
public class ExchangerDemo {
public static void main(String args[]) {
System.out.println("Main process started");
Exchanger<DataClass> exchanger = new Exchanger<DataClass>();
Loop loop1 = new Loop(1, "First", exchanger);
Loop loop2 = new Loop(2, "Second", exchanger);
Loop loop3 = new Loop(3, "Third", exchanger);
new Thread(loop1).start();
new Thread(loop2).start();
new Thread(loop3).start();
System.out.println("Main process ended");
}
}

For your dependency you should make three classes, and have three distinct Exchange objects (one in each). So thread1 would be between 1 and 2 (output of 1 to 2), thread 2's would be between 2 and 3 and thread 3's exhanger would be between itself and 1. Remember the exchanger's would guard only until it had its input from its feeder, to till it passes to its receiver.
Also synchronized is not as bad as the books make out. use it. Watch http://www.youtube.com/watch?v=WTVooKLLVT8
Also for reference Best way of running two threads alternatively?
Also why do you need three threads? Can you use a thread pool and have each task to the 3 things ?

Many ProducerS and many ConsumerS. Making the last producer alive killing the consumers

I have a standard producer consumer problem. Producer puts data into the stack(buffer) consumers take it.
I would like to have many producers and consumers.
the problem is I would like to make only the last living producer to be able to call b.stop()
for(int i = 0; i < 10; i++){
try{
// sleep((int)(Math.random() * 1));
}catch(Exception e){e.printStackTrace();}
b.put((int) (Math.random()* 10));
System.out.println("i = " + i);
}
b.stop();
so then I call b.stop() which changes running field in Buffer to false and notifiesAll()
End then I get:
i = 9 // number of iteration this is 10th iteration
Consumer 2.: no data to take. I wait. Memory: 0
Consumer 1.: no data to take. I wait. Memory: 0
Consumer 3.: no data to take. I wait. Memory: 0
they should die then, so I made method stop() but it did not work.
Code is running please check it
import java.util.Stack;
public class Buffer {
private static int SIZE = 4;
private int i;//number of elements in buffer
public Stack<Integer> stack;
private volatile boolean running;
public Buffer() {
stack = new Stack<>();
running = true;
i = 0;
}
synchronized public void put(int val){
while (i >= SIZE) {
try {
System.out.println("Buffer full, producer waits");
wait();
} catch (InterruptedException exc) {
exc.printStackTrace();
}
}
stack.push(val);//txt = s;
i++;
System.out.println("Producer inserted " + val + " memory: " + i);
if(i - 1 == 0)
notifyAll();
System.out.println(stack);
}
public synchronized Integer get(Consumer c) {
while (i == 0) {
try {
System.out.println(c + ": no data to take. I wait. Memory: " + i);
wait();
} catch (InterruptedException exc) {
exc.printStackTrace();
}
}
if(running){
int data = stack.pop();
i--;
System.out.println(c+ ": I took: " + data +" memory: " + i);
System.out.println(stack);
if(i + 1 == SIZE){//if the buffer was full so the producer is waiting
notifyAll();
System.out.println(c + "I notified producer about it");
}
return data;}
else
return null;
}
public boolean isEmpty(){
return i == 0;
}
public synchronized void stop(){//I THOUGH THIS WOULD FIX IT~!!!!!!!!!!!!!!
running = false;
notifyAll();
}
public boolean isRunning(){
return running;
}
}
public class Producer extends Thread {
private Buffer b;
public Producer(Buffer b) {
this.b = b;
}
public void run(){
for(int i = 0; i < 10; i++){
try{
// sleep((int)(Math.random() * 1));
}catch(Exception e){e.printStackTrace();}
b.put((int) (Math.random()* 10));
System.out.println("i = " + i);
}
b.stop();
}
}
public class Consumer extends Thread {
Buffer b;
int nr;
static int NR = 0;
public Consumer(Buffer b) {
this.b = b;
nr = ++NR;
}
public void run() {
Integer i = b.get(this);
while (i != null) {
System.out.println(nr + " I received : " + i);
i = b.get(this);
}
System.out.println("Consumer " + nr + " is dead");
}
public String toString() {
return "Consumer " + nr + ".";
}
}
public class Main {
public static void main(String[] args) {
Buffer b = new Buffer();
Producer p = new Producer(b);
Consumer c1 = new Consumer(b);
Consumer c2 = new Consumer(b);
Consumer c3 = new Consumer(b);
p.start();
c1.start();c2.start();c3.start();
}
}

What you have to realise is that your threads could be waiting in either of two locations:
In the wait loop with i == 0 - in which case notifyall will kick all of them out. However, if i is still 0 they will go straight back to waiting again.
Waiting for exclusive access to the object (i.e. waiting on a synchronized method) - in which case (if you fix issue 1 above and the lock will be released) they will go straight into a while (i == 0) loop.
I would suggest you change your while ( i == 0 ) loop to while ( running && i == 0 ). This should fix your problem. Since your running flag is (correctly) volatile all should tidily exit.

In your stop method, you set running to false, but your while loop is running as long as i == 0. Set i to something different than zero and it should fix it.
BTW, I don't understand why you have a running variable and a separate i variable, which is actually the variable keeping a thread running.

I would rethink your design. Classes should have a coherent set of responsibilities; making a class responsible for both consuming objects off the queue, while also being responsible for shutting down other consumers, seems to be something you'd want to seperate.

In answer to the to make only the last living producer to be able to call b.stop().
You should add an AtomicInteger to your Buffer containing the number of producers and make each producer call b.start() (which increments it) in its constructor.
That way you can decrement it in b.stop() and only when it has gone to zero should running be set to false.