Develop Reference

Threads, synchronization, filling and removing int's from an array

I need to write a program, which has an integer array. Lets say the array's length is 3. To start with, it is empty. Now, the program has to use two threads, one writes integers into the empty array, the other one deletes them out of it. I will give you an example:
thread1 put in value 1 //array has value: 1
thread1 put in value 2 //array has two values: 1 and 2
thread2 deleted value 2 //array now has only one value: 1
thread1 put in value 3 //array has two values: 1 and 3
thread1 put in value 4 //array has three values: 1,3,4
thread1 wants to put value 5 into the array, but it has to wait, because the array is full
thread2 deleted value 4// array has 2 values: 1, 3 and so on........
My teacher said, it would be best and easiest to do all of this by implementing for cycles with a specified counter(for example each thread has to go through 10 cycles). Now i have written some code, but i just cant figure out how to implement the synchronized object(the integer array) so that both threads can use it. Heres the code:
public class Antras {
public static void main(String[] args) {
System.out.println("Program starts working");
begin();
System.out.println("Program ends work");
}
public static void begin() {
synchronizationObject channel = new synchronizationObject();
try {
Thread putIn = new ReadThread(channel);
putIn.start();
Thread takeOut = new WriteThread(channel);
takeOut.start();
putIn.join();
takeOut.join();
System.out.println("main() ended work");
} catch (InterruptedException exc) {
System.out.println("Error " + exc);
}
}
}
class ReadThread extends Thread {
private synchronizationObject channel;
public ReadThread(synchronizationObject channel) {
this.channel = channel;
}
public void run() {
System.out.println("Thread " + this + "working");
channel.working = false;
System.out.println("Thread " + this + "ends work");
}
}
class WriteThread extends Thread {
private synchronizationObject channel;
public WriteThread(synchronizationObject channel) {
this.channel = channel;
}
public void run() {
System.out.println("Thread " + this + "working");
System.out.println("Thread " + this + "end work");
}
}
class synchronizationObject {
public static int N = 3;
int[] arrayOfInts = new int[N];
synchronized void takeOut() {
}
synchronized void putIn(int d) {
}
}

synchronized int putIn(int d) {
if(full == N){
try {
wait();
System.out.println("Array full, thread waits");
} catch (InterruptedException e) {
e.printStackTrace();
}
}else{
buffer[idxdeti] = d;
System.out.println("Value " + d + " added");
idxdeti++;
full++;
if (idxdeti == N){
idxdeti = 0;
}
}
return d; //returns the value
}
This is the WriteThread's run method:
public void run() {
System.out.println("WriteThread" + this + "started");
for(int i = 0; i<10; i++){
channel.putIn(i); //puts in a value
}
System.out.println("WriteThread" + this + "ended");
}
Now I just don't know how to return a value to the takeOut method, so when it works, it removes the given value.

Why is this program creating more threads than possible?

This is for a custom UDTF in a hive query, CreateLogTable is the UDTF class which I am using as a temp for testing. I am creating one thread per file to be downloaded from Amazon S3 and waiting until another thread becomes available before allocating another file to the thread.
Main Test logic:
CreateLogTable CLT = new CreateLogTable();
int numThreads = 2;
int index = 0;
DownloadFileThread[] dlThreads = new DownloadFileThread[numThreads];
for (S3ObjectSummary oSummary : bucketKeys.getObjectSummaries()) {
while (dlThreads[index] != null && dlThreads[index].isAlive()) {
index += 1;
index = index % numThreads;
}
dlThreads[index] = new DownloadFileThread(CLT , getBucket(oSummary.getBucketName() + "/"
+ oSummary.getKey()), getFile(oSummary.getKey()), index);
dlThreads[index].start();
index += 1;
index = index % numThreads;
}
Thread class (run() method):
try {
System.out.println("Creating thread " + this.threadnum);
this.fileObj = this.S3CLIENT.getObject(new GetObjectRequest(this.filePath, this.fileName));
this.fileIn = new Scanner(new GZIPInputStream(this.fileObj.getObjectContent()));
while (this.fileIn.hasNext()) {
this.parent.forwardToTable(fileIn.nextLine());
}
System.out.println("Finished " + this.threadnum);
} catch (Throwable e) {
System.out.println("Downloading of " + this.fileName + " failed.");
}
The while loop before the thread creation should be looping until it finds a null thread or a dead thread until it exits the loop, in which case a new thread will be created and started. Since I included logging to console, I am able to observe this process, but the output is unexpected:
Creating thread 0
Creating thread 1
Creating thread 0
Creating thread 1
Creating thread 0
Creating thread 1
Creating thread 0
...
Creating thread 1
Creating thread 0
Creating thread 1
Finished 0
Finished 1
Finished 1
Finished 0
Finished 1
Finished 1
...
Finished 0
Finished 1
Finished 0
Finished 1
The above is only the first few lines of output. The issue is that more than two threads are created before any threads complete their tasks.
Why is this happening and how can I fix this?

I reduced your code to this test case:
public class ThreadTest {
private static class SleepThread extends Thread {
private final int index;
SleepThread(int ii) { index = ii; }
#Override
public void run() {
System.out.println("Creating thread " + this.index);
try {
Thread.sleep(5_000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Finished " + this.index);
}
}
public static void main(String[] args) {
int numThreads = 2;
int index = 0;
SleepThread[] dlThreads = new SleepThread[numThreads];
for (int ii = 0; ii < 10; ++ii) {
while (dlThreads[index] != null && dlThreads[index].isAlive()) {
index += 1;
index = index % numThreads;
}
dlThreads[index] = new SleepThread(index);
dlThreads[index].start();
index += 1;
index = index % numThreads;
}
}
}
Using Sun JDK 1.7.0_75, running this produces the result that you'd expect--two threads start, they exit after five seconds, two more threads start, and so on.
The next thing I'd suspect is that your JVM's implementation of Thread.isAlive() isn't returning true for threads immediately after they are started, although that seems contrary to the documentation for the Thread class.

Try to see this example:
public static void main(String[] args) {
ExecutorService executor = Executors.newFixedThreadPool(5);
for (int i = 0; i < 10; i++) {
Runnable worker = new WorkerThread("" + i);
executor.execute(worker);
}
executor.shutdown();
while (!executor.isTerminated()) {
}
System.out.println("Finished all threads");
}
It's a thread pool using Java 8. A very simple and esay way to make it using the Executors. Very staraight forward way to make it.

The reason why the above code wasn't working was because of something wacky going on with the call to isAlive().
For some reason, no matter what state a thread is in, isAlive() will always return false for me, causing the creation of more and more threads, which replace the old ones in the array, dlThreads.
I solved the issue by creating a custom isWorking() method which simply returns a boolean of whether or not the thread's run() method has completed. Here is what the Thread class looks like now:
//this.isWorking initialized to true during instantiation
#Override
public void run() {
try {
System.out.println("Creating thread " + this.threadnum + " for " + filePath + "/" + fileName);
this.fileObj = this.S3CLIENT.getObject(new GetObjectRequest(this.filePath, this.fileName));
this.fileIn = new Scanner(new GZIPInputStream(this.fileObj.getObjectContent()));
while (this.fileIn.hasNext()) {
this.parent.forwardToTable(fileIn.nextLine());
}
System.out.println("Finished " + this.threadnum);
this.isWorking = false;
} catch (Throwable e) {
System.out.println("Downloading of " + this.fileName + " failed.");
e.printStackTrace();
this.isWorking = false;
}
}
public boolean isWorking(){
return this.isWorking;
}
However, after implementing this and being satisfied that my multithreaded script works, I switched over to using an Executor, as suggested by other users, which slightly improved performance and made the code much cleaner.

Java thread not responding to volatile boolean flag

I am new to Java concurrency, and I met a very strange problem:
I read from a large file and used several worker threads to work on the input (some complicated string matching tasks). I used a LinkedBlockingQueue to transmit the data to the worker threads, and a volatile boolean flag in the worker class to respond to the signal when the end-of-file is reached.
However, I cannot get the worker thread to stop properly. The CPU usage by this program is almost zero in the end, but the program won't terminate normally.
The simplified code is below. I have removed the real code and replaced them with a simple word counter. But the effect is the same. The worker thread won't terminate after the whole file is processed, and the boolean flag is set to true in the main thread.
The class with main
public class MultiThreadTestEntry
{
private static String inputFileLocation = "someFile";
private static int numbOfThread = 3;
public static void main(String[] args)
{
int i = 0;
Worker[] workers = new Worker[numbOfThread];
Scanner input = GetIO.getTextInput(inputFileLocation);
String temp = null;
ExecutorService es = Executors.newFixedThreadPool(numbOfThread);
LinkedBlockingQueue<String> dataQueue = new LinkedBlockingQueue<String>(1024);
for(i = 0 ; i < numbOfThread ; i ++)
{
workers[i] = new Worker(dataQueue);
workers[i].setIsDone(false);
es.execute(workers[i]);
}
try
{
while(input.hasNext())
{
temp = input.nextLine().trim();
dataQueue.put(temp);
}
}
catch (InterruptedException e)
{
Thread.currentThread().interrupt();
}
input.close();
for(i = 0 ; i < numbOfThread ; i ++)
{
workers[i].setIsDone(true);
}
es.shutdown();
try
{
es.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
} catch (InterruptedException e)
{
Thread.currentThread().interrupt();
}
}
}
The worker class
public class Worker implements Runnable
{
private LinkedBlockingQueue<String> dataQueue = null;
private volatile boolean isDone = false;
public Worker(LinkedBlockingQueue<String> dataQueue)
{
this.dataQueue = dataQueue;
}
#Override
public void run()
{
String temp = null;
long count = 0;
System.out.println(Thread.currentThread().getName() + " running...");
try
{
while(!isDone || !dataQueue.isEmpty())
{
temp = dataQueue.take();
count = temp.length() + count;
if(count%1000 == 0)
{
System.out.println(Thread.currentThread().getName() + " : " + count);
}
}
System.out.println("Final result: " + Thread.currentThread().getName() + " : " + count);
}
catch (InterruptedException e)
{
}
}
public void setIsDone(boolean isDone)
{
this.isDone = isDone;
}
}
Any suggestions to why this happens?
Thank you very much.

As Dan Getz already said your worker take() waits until an element becomes available but the Queue may be empty.
In your code you check if the Queue is empty but nothing prevents the other Workers to read and remove an element from the element after the check.
If the Queue contains only one element and t1 and t2 are two Threads
the following could happen:
t2.isEmpty(); // -> false
t1.isEmpty(); // -> false
t2.take(); // now the queue is empty
t1.take(); // wait forever
in this case t1 would wait "forever".
You can avoid this by using pollinstead of take and check if the result is null
public void run()
{
String temp = null;
long count = 0;
System.out.println(Thread.currentThread().getName() + " running...");
try
{
while(!isDone || !dataQueue.isEmpty())
{
temp = dataQueue.poll(2, TimeUnit.SECONDS);
if (temp == null)
// re-check if this was really the last element
continue;
count = temp.length() + count;
if(count%1000 == 0)
{
System.out.println(Thread.currentThread().getName() + " : " + count);
}
}
System.out.println("Final result: " + Thread.currentThread().getName() + " : " + count);
}
catch (InterruptedException e)
{
// here it is important to restore the interrupted flag!
Thread.currentThread().interrupt();
}
}

How to create a cyclic exchange of three threads?

How to create a cyclic exchange of three threads? That is: first thread must send data to second, second to third and third thread must send data to first.
I wrote some code, but threads exchange in random oder.
class DataClass {
int value;
String message;
DataClass(int v, String s) {
value = v;
message = s;
}
int getValue() {
return (value);
}
String getMassage() {
return (message);
}
}
class Loop implements Runnable {
int counter;
String name;
Exchanger<DataClass> exchanger;
Loop(int startValue, String id, Exchanger<DataClass> ex) {
counter = startValue;
name = id;
exchanger = ex;
System.out.println(name + ": created");
}
public void run() {
System.out.println(name + ": started");
DataClass data = new DataClass(counter, name);
for (int i = 0; i < 3; ++i) {
try {
DataClass newData = exchanger.exchange(data);
counter += newData.getValue();
System.out.println(name + ": from "
+ newData.getMassage() + ": data: "
+ newData.getValue() + ": state = " + counter);
} catch (InterruptedException e) {
System.err.println(e.toString());
}
}
System.out.println(name + ": ended");
}
}
public class ExchangerDemo {
public static void main(String args[]) {
System.out.println("Main process started");
Exchanger<DataClass> exchanger = new Exchanger<DataClass>();
Loop loop1 = new Loop(1, "First", exchanger);
Loop loop2 = new Loop(2, "Second", exchanger);
Loop loop3 = new Loop(3, "Third", exchanger);
new Thread(loop1).start();
new Thread(loop2).start();
new Thread(loop3).start();
System.out.println("Main process ended");
}
}

For your dependency you should make three classes, and have three distinct Exchange objects (one in each). So thread1 would be between 1 and 2 (output of 1 to 2), thread 2's would be between 2 and 3 and thread 3's exhanger would be between itself and 1. Remember the exchanger's would guard only until it had its input from its feeder, to till it passes to its receiver.
Also synchronized is not as bad as the books make out. use it. Watch http://www.youtube.com/watch?v=WTVooKLLVT8
Also for reference Best way of running two threads alternatively?
Also why do you need three threads? Can you use a thread pool and have each task to the 3 things ?

Many ProducerS and many ConsumerS. Making the last producer alive killing the consumers

I have a standard producer consumer problem. Producer puts data into the stack(buffer) consumers take it.
I would like to have many producers and consumers.
the problem is I would like to make only the last living producer to be able to call b.stop()
for(int i = 0; i < 10; i++){
try{
// sleep((int)(Math.random() * 1));
}catch(Exception e){e.printStackTrace();}
b.put((int) (Math.random()* 10));
System.out.println("i = " + i);
}
b.stop();
so then I call b.stop() which changes running field in Buffer to false and notifiesAll()
End then I get:
i = 9 // number of iteration this is 10th iteration
Consumer 2.: no data to take. I wait. Memory: 0
Consumer 1.: no data to take. I wait. Memory: 0
Consumer 3.: no data to take. I wait. Memory: 0
they should die then, so I made method stop() but it did not work.
Code is running please check it
import java.util.Stack;
public class Buffer {
private static int SIZE = 4;
private int i;//number of elements in buffer
public Stack<Integer> stack;
private volatile boolean running;
public Buffer() {
stack = new Stack<>();
running = true;
i = 0;
}
synchronized public void put(int val){
while (i >= SIZE) {
try {
System.out.println("Buffer full, producer waits");
wait();
} catch (InterruptedException exc) {
exc.printStackTrace();
}
}
stack.push(val);//txt = s;
i++;
System.out.println("Producer inserted " + val + " memory: " + i);
if(i - 1 == 0)
notifyAll();
System.out.println(stack);
}
public synchronized Integer get(Consumer c) {
while (i == 0) {
try {
System.out.println(c + ": no data to take. I wait. Memory: " + i);
wait();
} catch (InterruptedException exc) {
exc.printStackTrace();
}
}
if(running){
int data = stack.pop();
i--;
System.out.println(c+ ": I took: " + data +" memory: " + i);
System.out.println(stack);
if(i + 1 == SIZE){//if the buffer was full so the producer is waiting
notifyAll();
System.out.println(c + "I notified producer about it");
}
return data;}
else
return null;
}
public boolean isEmpty(){
return i == 0;
}
public synchronized void stop(){//I THOUGH THIS WOULD FIX IT~!!!!!!!!!!!!!!
running = false;
notifyAll();
}
public boolean isRunning(){
return running;
}
}
public class Producer extends Thread {
private Buffer b;
public Producer(Buffer b) {
this.b = b;
}
public void run(){
for(int i = 0; i < 10; i++){
try{
// sleep((int)(Math.random() * 1));
}catch(Exception e){e.printStackTrace();}
b.put((int) (Math.random()* 10));
System.out.println("i = " + i);
}
b.stop();
}
}
public class Consumer extends Thread {
Buffer b;
int nr;
static int NR = 0;
public Consumer(Buffer b) {
this.b = b;
nr = ++NR;
}
public void run() {
Integer i = b.get(this);
while (i != null) {
System.out.println(nr + " I received : " + i);
i = b.get(this);
}
System.out.println("Consumer " + nr + " is dead");
}
public String toString() {
return "Consumer " + nr + ".";
}
}
public class Main {
public static void main(String[] args) {
Buffer b = new Buffer();
Producer p = new Producer(b);
Consumer c1 = new Consumer(b);
Consumer c2 = new Consumer(b);
Consumer c3 = new Consumer(b);
p.start();
c1.start();c2.start();c3.start();
}
}

What you have to realise is that your threads could be waiting in either of two locations:
In the wait loop with i == 0 - in which case notifyall will kick all of them out. However, if i is still 0 they will go straight back to waiting again.
Waiting for exclusive access to the object (i.e. waiting on a synchronized method) - in which case (if you fix issue 1 above and the lock will be released) they will go straight into a while (i == 0) loop.
I would suggest you change your while ( i == 0 ) loop to while ( running && i == 0 ). This should fix your problem. Since your running flag is (correctly) volatile all should tidily exit.

In your stop method, you set running to false, but your while loop is running as long as i == 0. Set i to something different than zero and it should fix it.
BTW, I don't understand why you have a running variable and a separate i variable, which is actually the variable keeping a thread running.

I would rethink your design. Classes should have a coherent set of responsibilities; making a class responsible for both consuming objects off the queue, while also being responsible for shutting down other consumers, seems to be something you'd want to seperate.

In answer to the to make only the last living producer to be able to call b.stop().
You should add an AtomicInteger to your Buffer containing the number of producers and make each producer call b.start() (which increments it) in its constructor.
That way you can decrement it in b.stop() and only when it has gone to zero should running be set to false.