I am testing the use of semaphores with the typical producer-consumer problem where I only have one producer and one consumer. The producer adds products one at a time and the consumer can withdraw several simultaneously.
To perform the test, the producer and the consumer store and remove numbers from a array of 10 elements where 0 represents that there are no products and any other number represents a product. Access to store and retrieve items is centralized in a class called Data. I use a mutex to make an orderly use of the vector in case we have more than one thread working simultaneously.
When executing it, I observe that the number of permissions is not correct according to the operations performed by the threads. The application shows an error because the semaphore of the producer says that it has permission, but the data vector is full.
package producer.consumer;
import java.io.IOException;
public class ProducerConsumer {
public static void main(String[] args) throws IOException {
final int MAX = 10;
Data data = new Data(MAX);
Consumer consumer = new Consumer(data);
Producer producer = new Producer(data);
consumer.start();
producer.start();
}
}
package producer.consumer;
public class Producer extends Thread{
private final Data data;
public Producer(Data data) {
this.data = data;
}
#Override
public void run() {
while (true) {
try {
data.add((int) (Math.random() * data.getLength()) + 1);
} catch (InterruptedException ex) {
System.out.println(ex.getMessage());
}
}
}
}
package producer.consumer;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Consumer extends Thread{
private final Data data;
public Consumer(Data data) {
this.data = data;
}
#Override
public void run() {
while (true) {
try {
data.remove((int) (Math.random() * data.getLength()) + 1);
} catch (InterruptedException ex) {
Logger.getLogger(Consumer.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
}
package producer.consumer;
import java.io.IOException;
import java.util.Arrays;
import java.util.concurrent.Semaphore;
public class Data {
private final int[] data;
private final Semaphore mutex = new Semaphore(1);
private final Semaphore semProducer, semConsumer;
public Data(int MAX) throws IOException {
data = new int[MAX];
semProducer = new Semaphore(MAX);
semConsumer = new Semaphore(0);
}
public int getLength() {
return data.length;
}
public void add(int number) throws InterruptedException {
semProducer.acquire();
mutex.acquire();
System.out.println("trying to add a product");
int i = 0;
while (data[i] != 0) {
i++;
}
data[i] = number;
int permits = semConsumer.availablePermits() + 1;
System.out.println("data added in " + i + " " + Arrays.toString(data)
+ " Resources consumer " + permits
+ " Resources producer " + semProducer.availablePermits());
mutex.release();
semConsumer.release();
}
public void remove(int numberElements) throws InterruptedException {
semConsumer.acquire(numberElements);
mutex.acquire();
System.out.println("trying to withdraw " + numberElements);
for (int i = 0; i < numberElements; i++) {
if (data[i] != 0) {
data[i] = 0;
}
}
int permisos = semProducer.availablePermits() + 1;
System.out.println(" Retired " + numberElements + " " + Arrays.toString(data)
+ " Resources consumer " + semConsumer.availablePermits()
+ " Resources producer " + permisos);
mutex.release();
semProducer.release(numberElements);
}
}
Thank you very much for the help.
Your consumer does not always consume what it claims to consume.
for (int i = 0; i < numberElements; i++) {
if (data[i] != 0) {
data[i] = 0;
}
}
Suppose numberElements is 3, and that we have exactly 3 available elements in data[7], data[8], data[9].
The loop terminates with i == 3, nothing has been removed, but the producer semaphore will still be 'upped' by 3.
In the consumer, if you use i as the array index, it needs to cover the whole array, and you need a separate counter for 'elements removed'.
It is not the case that available elements will always be in the lowest-numbered data slots even though the producer fills those in first. Consider the time sequence that the producer manages to produce at least 5 elements, then the consumer runs to consume 2, and then immediately runs again to consume 3, before any more have been produced. data[0] and data[1] will be empty on the second run of the consumer and we run into the scenario I describe.
EDIT Acquiring and releasing permits seems correct; but you need to make sure that the consumer will actually clear the correct number of elements.
In example, edit the Data class with
public void remove(int numberElements) throws InterruptedException {
semConsumer.acquire(numberElements);
mutex.acquire();
System.out.println("remove: num-elem=" + numberElements);
int consumed=0;
for (int i = 0; consumed<numberElements; i++) {
if (data[i] != 0) {
data[i] = 0;
consumed++;
}
}
System.out.println(
" Retired " + numberElements + " " + Arrays.toString(data) );
mutex.release();
semProducer.release(numberElements);
}
Note also that this implementation is not very efficient (you'll need to iterate over the whole array both when inserting and deleting items, which can be expensive when MAX is large..)
Related
I need ships to load and unload cargo at the same time.
Is there a way to do that in java?
I managed to make multiple ships work in port at the same time, but they first unload cargo and then load new crates.
That is my variant of Ship class
public class Ship implements Runnable {
String name;
Port port;
Queue<Goods> storage;
Pier pier;
int capacity;
int numOnBoard;
public Ship(String name, Port port, int capacity) {
this.name = name;
this.port = port;
storage = new LinkedBlockingDeque<>(capacity);
this.capacity = capacity;
int num=(int)(Math.random()*capacity);
numOnBoard=num;
for (int i = 0; i < num; i++) {
storage.add(new Goods());
}
}
public void run() {
try {
int unl = 0;
int l = 0;
pier = port.getPier();
System.out.println("Ship " + name + " taken " + pier.name);
while (unload()) {
if(unl>=numOnBoard) break;
unl++;
System.out.println("Ship " + name + " unloaded cargo.");
Thread.sleep(new Random(100).nextInt(500));
}
System.out.println("Ship " + name + " unloaded " + unl + " crates.");
Thread.sleep(100);
while (load()) {
l++;
System.out.println("Ship " + name + " loaded cargo.");
Thread.sleep(new Random(100).nextInt(500));
}
System.out.println("Ship " + name + " loaded " + l + " crates.");
port.releasePier(pier);
System.out.println("Ship " + name + " released " + pier.name);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private boolean unload() {
if (storage.size() <= 0) return false;
return port.addGoods(storage.poll());
}
private boolean load() {
if (storage.size() >= capacity) return false;
return port.takeGoods(storage,numOnBoard);
}
}
And the Port
public class Port {
Queue<Pier> piers;
Queue<Goods> goods;
int capacity;
public Port(int pierCount, int capacity) {
goods = new LinkedBlockingDeque<>(capacity);
piers = new LinkedBlockingDeque<>(pierCount);
this.capacity = capacity;
for (int i = 0; i < pierCount; i++)
piers.add(new Pier("Pier " + (i + 1)));
int num=(int)(Math.random()*capacity);
for (int i = 0; i < num; i++) {
goods.add(new Goods());
}
}
public boolean addGoods(Goods item) {
if (goods.size() >= capacity) return false;
return goods.add(item);
}
public boolean takeGoods(Queue<Goods> storage, int wasOnBoard) {
if (goods.size() <= wasOnBoard) return false;
return storage.add(goods.poll());
}
public Pier getPier() {
Pier taken = piers.poll();
while (taken == null) {
try {
System.out.println("There aren't any free piers. Waiting...");
Thread.sleep(1000);
taken = piers.poll();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
return taken;
}
public void releasePier(Pier pier) {
piers.add(pier);
}
public static void main(String[] args) {
Port port = new Port(4, 50);
ArrayList<Thread> ships = new ArrayList<>();
for (int i = 0; i < 5; i++) {
ships.add(new Thread(new Ship("ship " + (i+1), port, 30)));
}
for (Thread t : ships)
t.start();
}
}
I need each ship to load and unload cargo at the same time
What you're trying to accomplish within a single thread is exactly what multiple threads are for. Multi-threading enables you to write in a way where multiple activities can proceed concurrently in the same program:
A multi-threaded program contains two or more parts that can run concurrently and each part can handle a different task at the same time making optimal use of the available resources specially when your computer has multiple CPUs.
By definition, multitasking is when multiple processes share common
processing resources such as a CPU. Multi-threading extends the idea
of multitasking into applications where you can subdivide specific
operations within a single application into individual threads. Each
of the threads can run in parallel. The OS divides processing time not
only among different applications, but also among each thread within
an application.
Read more about Java multi-threading here.
com.offbynull.coroutines version 1.1.0 consumers only consumes 7500 messages.
Please help me understand why this code only consumes 7500 messages instead of 30000.
public class DemoProducerConsumer {
public static int cnt = 0;
public static final int MAX = 10000;
public static class Producer implements Coroutine {
#Override
public void run(Continuation ctn) throws Exception {
String thName = Thread.currentThread().getName();
System.out.println(thName + ") Producer starting...");
Consumer consumer = new Consumer();
for (int i = 0; i < 3; i++) {
consumer.consume(ctn, "Hello:" + i);
}
System.out.println(thName + ") Producer published 3 messages");
}
}
public static class Consumer {
public void consume(Continuation ctn, String message) {
String thName = Thread.currentThread().getName();
System.out.println(thName + ")" + message);
cnt++; // <<< SUSPECT bug here.
ctn.suspend(); // <<< SUSPECT bug here.
}
}
public static final void main(String... args) throws InterruptedException {
String thName = Thread.currentThread().getName();
System.err.println(thName + ") Preparing Producer ");
new Thread(new Runnable() {
public void run() {
cnt = 0;
Producer producer = new Producer();
CoroutineRunner runner = new CoroutineRunner(producer);
for (int i = 0; i < MAX; i++) {
runner.execute();
}
System.out.println(thName + ") Producer Looped " + MAX + " times.");
}
}).start();
System.err.println(thName + ") Waiting " + (MAX * 3) + " message to be consumed...");
Thread.sleep(10000);
System.err.println(thName + ") Message consumed:" + cnt);
System.err.println(thName + ") Exiting...");
}
}
I plan to use this with Thread Pool to implement a higher performance MVC server.
Separation of consumer and producer is a must.
Author of coroutines here. You seem to be misunderstanding how the execute() method works. Everytime you call suspend(), execute() will return. When you call execute() again, it'll continue executing the method from the point which you suspended.
So, if you want to completely execute your coroutine MAX times, you need to change your main loop to the following:
for (int i = 0; i < MAX; i++) {
boolean stillExecuting;
do {
stillExecuting = runner.execute();
} while (stillExecuting);
}
In addition to that, since you're accessing the field cnt from separate threads, you should probably be marking cnt as volatile:
public static volatile int cnt = 0;
Running with the above changes produces what you expect for your output:
main) Producer Looped 10000 times.
main) Message consumed:30000
main) Exiting...
Also, you should spend some time evaluating whether coroutines are a good fit for your usecase. I don't understand the problem you're trying to solve, but it sounds like normal Java threading constructs may be a better fit.
I am new to Java concurrency, and I met a very strange problem:
I read from a large file and used several worker threads to work on the input (some complicated string matching tasks). I used a LinkedBlockingQueue to transmit the data to the worker threads, and a volatile boolean flag in the worker class to respond to the signal when the end-of-file is reached.
However, I cannot get the worker thread to stop properly. The CPU usage by this program is almost zero in the end, but the program won't terminate normally.
The simplified code is below. I have removed the real code and replaced them with a simple word counter. But the effect is the same. The worker thread won't terminate after the whole file is processed, and the boolean flag is set to true in the main thread.
The class with main
public class MultiThreadTestEntry
{
private static String inputFileLocation = "someFile";
private static int numbOfThread = 3;
public static void main(String[] args)
{
int i = 0;
Worker[] workers = new Worker[numbOfThread];
Scanner input = GetIO.getTextInput(inputFileLocation);
String temp = null;
ExecutorService es = Executors.newFixedThreadPool(numbOfThread);
LinkedBlockingQueue<String> dataQueue = new LinkedBlockingQueue<String>(1024);
for(i = 0 ; i < numbOfThread ; i ++)
{
workers[i] = new Worker(dataQueue);
workers[i].setIsDone(false);
es.execute(workers[i]);
}
try
{
while(input.hasNext())
{
temp = input.nextLine().trim();
dataQueue.put(temp);
}
}
catch (InterruptedException e)
{
Thread.currentThread().interrupt();
}
input.close();
for(i = 0 ; i < numbOfThread ; i ++)
{
workers[i].setIsDone(true);
}
es.shutdown();
try
{
es.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
} catch (InterruptedException e)
{
Thread.currentThread().interrupt();
}
}
}
The worker class
public class Worker implements Runnable
{
private LinkedBlockingQueue<String> dataQueue = null;
private volatile boolean isDone = false;
public Worker(LinkedBlockingQueue<String> dataQueue)
{
this.dataQueue = dataQueue;
}
#Override
public void run()
{
String temp = null;
long count = 0;
System.out.println(Thread.currentThread().getName() + " running...");
try
{
while(!isDone || !dataQueue.isEmpty())
{
temp = dataQueue.take();
count = temp.length() + count;
if(count%1000 == 0)
{
System.out.println(Thread.currentThread().getName() + " : " + count);
}
}
System.out.println("Final result: " + Thread.currentThread().getName() + " : " + count);
}
catch (InterruptedException e)
{
}
}
public void setIsDone(boolean isDone)
{
this.isDone = isDone;
}
}
Any suggestions to why this happens?
Thank you very much.
As Dan Getz already said your worker take() waits until an element becomes available but the Queue may be empty.
In your code you check if the Queue is empty but nothing prevents the other Workers to read and remove an element from the element after the check.
If the Queue contains only one element and t1 and t2 are two Threads
the following could happen:
t2.isEmpty(); // -> false
t1.isEmpty(); // -> false
t2.take(); // now the queue is empty
t1.take(); // wait forever
in this case t1 would wait "forever".
You can avoid this by using pollinstead of take and check if the result is null
public void run()
{
String temp = null;
long count = 0;
System.out.println(Thread.currentThread().getName() + " running...");
try
{
while(!isDone || !dataQueue.isEmpty())
{
temp = dataQueue.poll(2, TimeUnit.SECONDS);
if (temp == null)
// re-check if this was really the last element
continue;
count = temp.length() + count;
if(count%1000 == 0)
{
System.out.println(Thread.currentThread().getName() + " : " + count);
}
}
System.out.println("Final result: " + Thread.currentThread().getName() + " : " + count);
}
catch (InterruptedException e)
{
// here it is important to restore the interrupted flag!
Thread.currentThread().interrupt();
}
}
How to create a cyclic exchange of three threads? That is: first thread must send data to second, second to third and third thread must send data to first.
I wrote some code, but threads exchange in random oder.
class DataClass {
int value;
String message;
DataClass(int v, String s) {
value = v;
message = s;
}
int getValue() {
return (value);
}
String getMassage() {
return (message);
}
}
class Loop implements Runnable {
int counter;
String name;
Exchanger<DataClass> exchanger;
Loop(int startValue, String id, Exchanger<DataClass> ex) {
counter = startValue;
name = id;
exchanger = ex;
System.out.println(name + ": created");
}
public void run() {
System.out.println(name + ": started");
DataClass data = new DataClass(counter, name);
for (int i = 0; i < 3; ++i) {
try {
DataClass newData = exchanger.exchange(data);
counter += newData.getValue();
System.out.println(name + ": from "
+ newData.getMassage() + ": data: "
+ newData.getValue() + ": state = " + counter);
} catch (InterruptedException e) {
System.err.println(e.toString());
}
}
System.out.println(name + ": ended");
}
}
public class ExchangerDemo {
public static void main(String args[]) {
System.out.println("Main process started");
Exchanger<DataClass> exchanger = new Exchanger<DataClass>();
Loop loop1 = new Loop(1, "First", exchanger);
Loop loop2 = new Loop(2, "Second", exchanger);
Loop loop3 = new Loop(3, "Third", exchanger);
new Thread(loop1).start();
new Thread(loop2).start();
new Thread(loop3).start();
System.out.println("Main process ended");
}
}
For your dependency you should make three classes, and have three distinct Exchange objects (one in each). So thread1 would be between 1 and 2 (output of 1 to 2), thread 2's would be between 2 and 3 and thread 3's exhanger would be between itself and 1. Remember the exchanger's would guard only until it had its input from its feeder, to till it passes to its receiver.
Also synchronized is not as bad as the books make out. use it. Watch http://www.youtube.com/watch?v=WTVooKLLVT8
Also for reference Best way of running two threads alternatively?
Also why do you need three threads? Can you use a thread pool and have each task to the 3 things ?
I have a standard producer consumer problem. Producer puts data into the stack(buffer) consumers take it.
I would like to have many producers and consumers.
the problem is I would like to make only the last living producer to be able to call b.stop()
for(int i = 0; i < 10; i++){
try{
// sleep((int)(Math.random() * 1));
}catch(Exception e){e.printStackTrace();}
b.put((int) (Math.random()* 10));
System.out.println("i = " + i);
}
b.stop();
so then I call b.stop() which changes running field in Buffer to false and notifiesAll()
End then I get:
i = 9 // number of iteration this is 10th iteration
Consumer 2.: no data to take. I wait. Memory: 0
Consumer 1.: no data to take. I wait. Memory: 0
Consumer 3.: no data to take. I wait. Memory: 0
they should die then, so I made method stop() but it did not work.
Code is running please check it
import java.util.Stack;
public class Buffer {
private static int SIZE = 4;
private int i;//number of elements in buffer
public Stack<Integer> stack;
private volatile boolean running;
public Buffer() {
stack = new Stack<>();
running = true;
i = 0;
}
synchronized public void put(int val){
while (i >= SIZE) {
try {
System.out.println("Buffer full, producer waits");
wait();
} catch (InterruptedException exc) {
exc.printStackTrace();
}
}
stack.push(val);//txt = s;
i++;
System.out.println("Producer inserted " + val + " memory: " + i);
if(i - 1 == 0)
notifyAll();
System.out.println(stack);
}
public synchronized Integer get(Consumer c) {
while (i == 0) {
try {
System.out.println(c + ": no data to take. I wait. Memory: " + i);
wait();
} catch (InterruptedException exc) {
exc.printStackTrace();
}
}
if(running){
int data = stack.pop();
i--;
System.out.println(c+ ": I took: " + data +" memory: " + i);
System.out.println(stack);
if(i + 1 == SIZE){//if the buffer was full so the producer is waiting
notifyAll();
System.out.println(c + "I notified producer about it");
}
return data;}
else
return null;
}
public boolean isEmpty(){
return i == 0;
}
public synchronized void stop(){//I THOUGH THIS WOULD FIX IT~!!!!!!!!!!!!!!
running = false;
notifyAll();
}
public boolean isRunning(){
return running;
}
}
public class Producer extends Thread {
private Buffer b;
public Producer(Buffer b) {
this.b = b;
}
public void run(){
for(int i = 0; i < 10; i++){
try{
// sleep((int)(Math.random() * 1));
}catch(Exception e){e.printStackTrace();}
b.put((int) (Math.random()* 10));
System.out.println("i = " + i);
}
b.stop();
}
}
public class Consumer extends Thread {
Buffer b;
int nr;
static int NR = 0;
public Consumer(Buffer b) {
this.b = b;
nr = ++NR;
}
public void run() {
Integer i = b.get(this);
while (i != null) {
System.out.println(nr + " I received : " + i);
i = b.get(this);
}
System.out.println("Consumer " + nr + " is dead");
}
public String toString() {
return "Consumer " + nr + ".";
}
}
public class Main {
public static void main(String[] args) {
Buffer b = new Buffer();
Producer p = new Producer(b);
Consumer c1 = new Consumer(b);
Consumer c2 = new Consumer(b);
Consumer c3 = new Consumer(b);
p.start();
c1.start();c2.start();c3.start();
}
}
What you have to realise is that your threads could be waiting in either of two locations:
In the wait loop with i == 0 - in which case notifyall will kick all of them out. However, if i is still 0 they will go straight back to waiting again.
Waiting for exclusive access to the object (i.e. waiting on a synchronized method) - in which case (if you fix issue 1 above and the lock will be released) they will go straight into a while (i == 0) loop.
I would suggest you change your while ( i == 0 ) loop to while ( running && i == 0 ). This should fix your problem. Since your running flag is (correctly) volatile all should tidily exit.
In your stop method, you set running to false, but your while loop is running as long as i == 0. Set i to something different than zero and it should fix it.
BTW, I don't understand why you have a running variable and a separate i variable, which is actually the variable keeping a thread running.
I would rethink your design. Classes should have a coherent set of responsibilities; making a class responsible for both consuming objects off the queue, while also being responsible for shutting down other consumers, seems to be something you'd want to seperate.
In answer to the to make only the last living producer to be able to call b.stop().
You should add an AtomicInteger to your Buffer containing the number of producers and make each producer call b.start() (which increments it) in its constructor.
That way you can decrement it in b.stop() and only when it has gone to zero should running be set to false.