I'm currently working on java application which has a scenario of multiple producers adding tasks to a queue and whenever queue is not empty tasks should be executed at predefined rate. (using multiple threads to maintain execution rate) After executing the available tasks executor has to wait till tasks available in the queue again.
I know blockingQueue can be used to triggering part in here and ScheduledExecutorService for execute tasks at fixed rate. But I could not find a way to link ability of both of this for my need. So I would be very thankful if you could give me any suggestion to make this happen.
You need the task queue to be accessible by both the producer and consumer threads. I've written a basic program to demonstrate this, but I'll let you play around with the BlockingQueue API and the ScheduledExecutor as per your needs:
import java.util.concurrent.*;
public class ProducerConsumer {
private static final BlockingQueue<Integer> taskQueue = new LinkedBlockingQueue<>();
public static void main(String[] args) {
ExecutorService consumers = Executors.newFixedThreadPool(3);
consumers.submit(new Consumer());
consumers.submit(new Consumer());
consumers.submit(new Consumer());
ExecutorService producers = Executors.newFixedThreadPool(2);
producers.submit(new Producer(1));
producers.submit(new Producer(2));
}
private static class Producer implements Runnable {
private final int task;
Producer(int task) {
this.task = task;
}
#Override
public void run() {
System.out.println("Adding task: " + task);
taskQueue.add(task); // put is better, since it will block if queue is full
}
}
private static class Consumer implements Runnable {
#Override
public void run() {
try {
Integer task = taskQueue.take(); // block if there is no task available
System.out.println("Executing task: " + task);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
This is the way I could come up with as a solution. It looks little bit rusty but I have tested this and the code is working.
package test;
import java.util.concurrent.*;
public class FixedRateConsumer {
private BlockingQueue<String> queue = new ArrayBlockingQueue<>(20);
private ScheduledExecutorService executorService = new ScheduledThreadPoolExecutor(5);
private boolean continueRunning = true;
public void executeInBackGraound() throws InterruptedException, ExecutionException {
while (continueRunning) {
String s = queue.take();
Worker w = new Worker(s);
ScheduledFuture future = executorService.scheduleAtFixedRate(w, 0, 1, TimeUnit.SECONDS);
w.future = future;
try {
if (!future.isDone()) {
future.get();
}
} catch (CancellationException e) {
// Skipping
}
}
}
public void setContinueRunning(boolean state) {
continueRunning = state;
}
public void addConsumableObject(String s) throws InterruptedException {
queue.put(s);
}
private void consumeString(String s) {
System.out.println("Consumed -> " + s + ", ... # -> " + System.currentTimeMillis() + " ms");
}
private class Worker implements Runnable {
String consumableObject;
ScheduledFuture future;
public Worker(String initialConsumableObject) {
this.consumableObject = initialConsumableObject;
}
#Override
public void run() {
try {
if (consumableObject == null) {
consumableObject = queue.take();
}
consumeString(consumableObject);
consumableObject = null;
if (queue.isEmpty()) {
if (future == null) {
while (future == null) {
Thread.sleep(50);
}
}
future.cancel(false);
}
} catch (Exception e) {
System.out.println("Exception : " + e);
}
}
}
}
Related
I am trying to implement a small producer-consumer example in Java using ExecutorService.
Here is my main class
class Example {
public static void main(String args[]) {
BlockingQueue<String> queue = new ArrayBlockingQueue<>(1000);
Producer producer = new Producer(queue);
Consumer consumer = new Consumer(queue);
ExecutorService executor = Executors.newCachedThreadPool();
// executor.execute(consumer);
Future producerFuture = executor.submit(producer);
Future consumerFuture = executor.submit(consumer);
try {
producerFuture.get();
consumerFuture.get();
} catch (InterruptedException e) {
LOG.error("Failed");
}
executor.shutdown();
executor.awaitTermination(10, TimeUnit.MILLISECONDS);
}
}
Producer Class
public class Producer implements Runnable {
private BlockingQueue<String> queue;
public Producer(BlockingQueue<String> queue) {
this.queue = queue;
}
#Override
public void run() {
for (int i = 0; i < 10; i++) {
try {
queue.put(i + "HELPPPPP");
} catch (InterruptedException ex) {
Logger.getLogger(MigrationToolProducer.class.getName()).log(Level.SEVERE, null, ex);
}
}
Consumer Class
public class Consumer implements Runnable {
private final BlockingQueue<String> queue;
private volatile boolean keepRunning = true;
public Consumer(BlockingQueue<String> queue) {
this.queue = queue;
}
#Override
public void run() {
while (keepRunning) {
String value;
try {
value = queue.take();
} catch(InterruptedException e) {
throw new RuntimeException(e);
}
System.out.println(value);
}
}
}
EDIT The execution is stuck at queue.take() in Consumer Class.
Can anyone please help me fix this problem ? Why is the execution stuck in the consumer ?
One possible solution:
1) On Producer side, put a "END" signal after original 10 puts:
queue.put("END");
2) On Consumer side, once detect "END" signal, break the loop:
public void run() {
while (keepRunning) {
String value;
try {
value = queue.take();
if(value.equals("END")) {
System.out.println("Get END signal. All done!");
break;
}
} catch(InterruptedException e) {
throw new RuntimeException(e);
}
System.out.println(value);
}
}
What is the best practice approach to launch a pool of 1000's of tasks (where up to 4 should be able to execute in parallel) and automatically timeout them if they take more than 3 seconds (individually)?
While I found that ExecutorService seems to be helpful (see SSCE from another post below), I don't see how to make this work for multiple tasks running in parallel (as the future.get(3, TimeUnit.SECONDS) is executing on the same thread than the one launching the tasks, hence no opportunity to launch multiple tasks in parallel):
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
public class Test {
public static void main(String[] args) throws Exception {
ExecutorService executor = Executors.newSingleThreadExecutor();
Future<String> future = executor.submit(new Task());
try {
System.out.println("Started..");
System.out.println(future.get(3, TimeUnit.SECONDS));
System.out.println("Finished!");
} catch (TimeoutException e) {
future.cancel(true);
System.out.println("Terminated!");
}
executor.shutdownNow();
}
}
class Task implements Callable<String> {
#Override
public String call() throws Exception {
Thread.sleep(4000); // Just to demo a long running task of 4 seconds.
return "Ready!";
}
}
Thanks!
If you have to monitor each task to kill it when it exceeds the timeout period, either
the task itself has to keep track of time and quit appropriately, OR
you have to create a second watchdog thread for every task. The watchdog thread sets a timer and sleeps, waking up after the timeout interval expires and then terminating the task if it's still running.
This is a tricky one. Here’s what I came up with:
public class TaskQueue<T> {
private static final Logger logger =
Logger.getLogger(TaskQueue.class.getName());
private final Collection<Callable<T>> tasks;
private final int maxTasks;
private int addsPending;
private final Collection<T> results = new ArrayList<T>();
private final ScheduledExecutorService executor;
public TaskQueue() {
this(4);
}
public TaskQueue(int maxSimultaneousTasks) {
maxTasks = maxSimultaneousTasks;
tasks = new ArrayDeque<>(maxTasks);
executor = Executors.newScheduledThreadPool(maxTasks * 3);
}
private void addWhenAllowed(Callable<T> task)
throws InterruptedException,
ExecutionException {
synchronized (tasks) {
while (tasks.size() >= maxTasks) {
tasks.wait();
}
tasks.add(task);
if (--addsPending <= 0) {
tasks.notifyAll();
}
}
Future<T> future = executor.submit(task);
executor.schedule(() -> future.cancel(true), 3, TimeUnit.SECONDS);
try {
T result = future.get();
synchronized (tasks) {
results.add(result);
}
} catch (CancellationException e) {
logger.log(Level.FINE, "Canceled", e);
} finally {
synchronized (tasks) {
tasks.remove(task);
if (tasks.isEmpty()) {
tasks.notifyAll();
}
}
}
}
public void add(Callable<T> task) {
synchronized (tasks) {
addsPending++;
}
executor.submit(new Callable<Void>() {
#Override
public Void call()
throws InterruptedException,
ExecutionException {
addWhenAllowed(task);
return null;
}
});
}
public Collection<T> getAllResults()
throws InterruptedException {
synchronized (tasks) {
while (addsPending > 0 || !tasks.isEmpty()) {
tasks.wait();
}
return new ArrayList<T>(results);
}
}
public void shutdown() {
executor.shutdown();
}
}
I suspect it could be done more cleanly using Locks and Conditions instead of synchronization.
I have a worker threadpool set up that executes a bit of work which I want to log in a central place.
To be more precise, I've extended the Thread class into a worker class, which checks the status of a concurrent queue. If it's empty, then it waits. As elements are added by another thread, notify() wakes the workers. Once they've completed the task, they wait for the next element in the queue.
What's the best practice to have each of the threads report their status at the end of each of their tasks?
public class PoolWorker extends Thread {
public ConcurrentLinkedQueue<Device> q;
public PoolWorker(ConcurrentLinkedQueue<Device> q, String type){
this.q = q;
this.type = type;
}
#Override
public void run(){
while (true)
{
Device d = null;
try{
synchronized(q){
while(q.isEmpty())
{
q.wait(); // wait for a notify()
}
d = q.remove();
}
// do some work
// report status of work completed
}
}
Try to do something like this
ExecutorService exec = Executors.newFixedThreadPool(10);
Runnable runn = new Runnable()
{
#Override
public void run()
{
System.out.println("");
}
};
exec.execute(runn);
As mentioned best way is to use BlockingQueue. Below is the sample code:
public class PoolWorker extends Thread {
public ArrayBlockingQueue<String> q;
public String type;
public PoolWorker(ArrayBlockingQueue<String> q, String type) {
this.q = q;
this.type = type;
}
#Override
public void run() {
while(true){
String work = null;
try {
System.out.println("PoolWorker.run:waiting .............");
work = q.take();
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("PoolWorker.run..work: " + work);
}
}
public static void main(String[] args) throws InterruptedException {
ArrayBlockingQueue<String> pool = new ArrayBlockingQueue<String>(100);
PoolWorker worker = new PoolWorker(pool, "Something");
worker.start();
addWork(pool, "work1");
addWork(pool, "work2");
addWork(pool, "work3");
addWork(pool, "work4");
addWork(pool, "work5");
//Just give enough time to run
Thread.sleep(5000);
}
private static void addWork(ArrayBlockingQueue<String> pool, String work) throws InterruptedException {
System.out.println("PoolWorker.addWork: " + work);
pool.put(work);
}
}
There is nice sample code available in Java documentation as well:
http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/BlockingQueue.html
Problem description : -
Step 1: Take input FILE_NAME from user at main thread.
Step 2: Perform 10 operations on that file (i.e count chars, count lines etc.. ), and all those 10 operations must be in septate threads. It means there must be 10 child threads.
Step 3: Main thread waits until all those child threads completed.
Step 4: Print result.
What I did :-
I did a sample code with 3 threads. I don't want file operation code from your side.
public class ThreadTest {
// This is object to synchronize on.
private static final Object waitObject = ThreadTest.class;
// Your boolean.
private static boolean boolValue = false;
public final Result result = new Result();
public static void main(String[] args) {
final ThreadTest mytest = new ThreadTest();
System.out.println("main started");
new Thread(new Runnable() {
public void run() {
System.out.println("Inside thread");
//Int initialiser
new Thread(new Runnable() {
public void run() {
System.out.println("Setting integer value");
mytest.result.setIntValue(346635);
System.out.println("Integer value seted");
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}).start();
//String initialiser
new Thread(new Runnable() {
public void run() {
System.out.println("Setting string value");
mytest.result.setStringValue("Hello hi");
System.out.println("String value seted");
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}).start();
//Boolean initialiser
new Thread(new Runnable() {
public void run() {
System.out.println("Setting boolean value");
mytest.result.setBoolValue(true);
System.out.println("Boolean value seted");
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}).start();
System.out.println("Thread is finished");
//Notify to main thread
synchronized (ThreadTest.waitObject) {
ThreadTest.boolValue = true;
ThreadTest.waitObject.notifyAll();
}
}
}).start();
try {
synchronized (ThreadTest.waitObject) {
while (!ThreadTest.boolValue) {
ThreadTest.waitObject.wait();
}
}
} catch (InterruptedException ie) {
ie.printStackTrace();
}
System.out.println("main finished");
System.out.println("Result is : " + mytest.result.toString());
}
}
Problem :-
My above code is not giving correct answer. How can I do that?
Alternate solutions:
CountDownLatch class does the same. But I don't want to use that class.
I looked this similar solution and I want to use methods of Thread only.
You can do:
Thread t = new Thread() {
public void run() {
System.out.println("text");
// other complex code
}
};
t.start();
t.join();
This way you will wait until the thread finishes and just then continue. You can join multiple threads:
for (Thread thread : threads) {
thread.join();
}
I would recommend looking at the Executors framework first, and then look into the CompletionService.
Then you can write something like this:
ExecutorService executor = Executors.newFixedThreadPool(maxThreadsToUse);
CompletionService completion = new ExecutorCompletionService(executor);
for (each sub task) {
completion.submit(new SomeTaskYouCreate())
}
// wait for all tasks to complete.
for (int i = 0; i < numberOfSubTasks; ++i) {
completion.take(); // will block until the next sub task has completed.
}
executor.shutdown();
In Java 8 a far better approach is to use parallelStream()
Note: it is far easier to see exactly what these background tasks are doing.
public static void main(String[] args) {
Stream.<Runnable>of(
() -> mytest.result.setIntValue(346635),
() -> mytest.result.setStringValue("Hello hi"),
() -> mytest.result.setBoolValue(true) )
.parallel()
.forEach(Runnable::run);
System.out.println("main finished");
System.out.println("Result is : " + mytest.result.toString());
}
I took out the debug information and the sleep as these don't alter the outcome.
You may want to choose CountDownLatch from java.util.concurrent. From JavaDocs:
A synchronization aid that allows one or more threads to wait until a
set of operations being performed in other threads completes.
Sample code:
import java.util.concurrent.CountDownLatch;
public class Test {
private final ChildThread[] children;
private final CountDownLatch latch;
public Test() {
this.children = new ChildThread[4];
this.latch = new CountDownLatch(children.length);
children[0] = new ChildThread(latch, "Task 1");
children[1] = new ChildThread(latch, "Task 2");
children[2] = new ChildThread(latch, "Task 3");
children[3] = new ChildThread(latch, "Task 4");
}
public void run() {
startChildThreads();
waitForChildThreadsToComplete();
}
private void startChildThreads() {
Thread[] threads = new Thread[children.length];
for (int i = 0; i < threads.length; i++) {
ChildThread child = children[i];
threads[i] = new Thread(child);
threads[i].start();
}
}
private void waitForChildThreadsToComplete() {
try {
latch.await();
System.out.println("All child threads have completed.");
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private class ChildThread implements Runnable {
private final String name;
private final CountDownLatch latch;
protected ChildThread(CountDownLatch latch, String name) {
this.latch = latch;
this.name = name;
}
#Override
public void run() {
try {
// Implementation
System.out.println(name + " has completed.");
} finally {
latch.countDown();
}
}
}
public static void main(String[] args) {
Test test = new Test();
test.run();
}
}
Output:
Task 1 has completed.
Task 4 has completed.
Task 3 has completed.
Task 2 has completed.
All child threads have completed.
There are many ways to approach this. Consider CountDownLatch:
import java.util.concurrent.CountDownLatch;
public class WorkerTest {
final int NUM_JOBS = 3;
final CountDownLatch countDownLatch = new CountDownLatch(NUM_JOBS);
final Object mutex = new Object();
int workData = 0;
public static void main(String[] args) throws Exception {
WorkerTest workerTest = new WorkerTest();
workerTest.go();
workerTest.awaitAndReportData();
}
private void go() {
for (int i = 0; i < NUM_JOBS; i++) {
final int fI = i;
Thread t = new Thread() {
public void run() {
synchronized(mutex) {
workData++;
}
try {
Thread.sleep(fI * 1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
countDownLatch.countDown();
}
};
t.start();
}
}
private void awaitAndReportData() throws InterruptedException {
countDownLatch.await();
synchronized(mutex) {
System.out.println("All workers done. workData=" + workData);
}
}
}
Check if all child threads are dead, every n seconds. Simple, yet effective method:
boolean allDead=false;
while(! allDead){
allDead=true;
for (int t = 0; t < threadCount; t++)
if(threads[t].isAlive()) allDead=false;
Thread.sleep(2000);
}
I tried to write a file monitor which will check the file if a new line is appended,the monitor in fact is a thread which will read the line by a randomaccessfile all the time.
This is the monitor core codes:
public class Monitor {
public static Logger log = Logger.getLogger(Monitor.class);
public static final Monitor instance = new Monitor();
private static final ArrayList<Listener> registers = new ArrayList<Listener>();
private Runnable task = new MonitorTask();
private Thread monitorThread = new Thread(task);
private boolean beStart = true;
private static RandomAccessFile raf = null;
private File monitoredFile = null;
private long lastPos;
public void register(File f, Listener listener) {
this.monitoredFile = f;
registers.add(listener);
monitorThread.start();
}
public void replaceFile(File newFileToBeMonitored) {
this.monitoredFile = newFileToBeMonitored;
// here,how to restart the monitorThread?
}
private void setRandomFile() {
if (!monitoredFile.exists()) {
log.warn("File [" + monitoredFile.getAbsolutePath()
+ "] not exist,will try again after 30 seconds");
try {
Thread.sleep(30 * 1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
setRandomFile();
return;
}
try {
if (raf != null) {
raf.close();
lastPos = 0;
}
raf = new RandomAccessFile(monitoredFile, "r");
log.info("monitor file " + monitoredFile.getAbsolutePath());
} catch (FileNotFoundException e) {
// The file must exist now
} catch (IOException e) {}
}
private void startRead() {
beStart = true;
String line;
while (beStart) {
try {
raf.seek(lastPos);
while ((line = raf.readLine()) != null) {
fireEvent(new FileEvent(monitoredFile.getAbsolutePath(),
line));
}
lastPos = raf.getFilePointer();
} catch (IOException e1) {}
}
}
private void stopRead() {
this.beStart = false;
}
private void fireEvent(FileEvent event) {
for (Listener lis : registers) {
lis.lineAppended(event);
}
}
private class MonitorTask implements Runnable {
#Override
public void run() {
stopRead();
//why putting the resetReandomAccessFile in this thread method is that it will sleep if the file not exist.
setRandomFile();
startRead();
}
}
}
This is some help classes:
public interface Listener {
void lineAppended(FileEvent event);
}
public class FileEvent {
private String line;
private String source;
public FileEvent(String filepath, String addedLine) {
this.line = addedLine;
this.source = filepath;
}
//getter and setter
}
And this is a example to call the monitor:
public class Client implements Listener {
private static File f = new File("D:/ab.txt");
public static void main(String[] args) {
Monitor.instance.register(f, new Client());
System.out.println(" I am done in the main method");
try {
Thread.sleep(5000);
Monitor.instance.replaceFile(new File("D:/new.txt"));
} catch (InterruptedException e) {
System.out.println(e.getMessage());
}
}
#Override
public void lineAppended(FileEvent event) {
String line = event.getLine();
if (line.length() <= 0)
return;
System.err.println("found in listener:" + line + ":" + line.length());
}
}
Now,my probelm is the code work well if I just call:
Monitor.instance.register(file,listener);
This will monitor the file for line appending,and will notify the listener.
However it does not work when I call the :
Monitor.instance.replaceFile(anotherfile);
This means I want to monitor another file rather than before.
So in my Monitor I have to restart the thread,how to make it?
I have tried the:
monitorThread.interruppt();
It does not wrok.
Anyone can fix it for me or tell me how to do ?
Thanks.
Before I ask,I have googling the "restart java thread",so I know one can not restart a dead thread,but my thread does not return,so I think it can be restarted.
You don't restart a Thread, instead you create a new one each time you want to start a thread.
A better alternative may be to use Executors.newCachedThreadPool() which gives you a pool of thread which will be started/recycle for you.
BTW: You are using recursion rather than a loop to poll if the file exists. Using recursion can mean if you wait too long it will throw a StackOverflowError. IMHO you shouldn't wait at all, the polling thread should repeatedly attempt to open the file until it is told to stop (or the file appears)
Your current implementation also means if the file is replaced, you will have to reopen the file in the background thread anyway.
Instead of explaining, I just coded up a skeleton example. I did not test it terribly well, but it may be of some use.
In order to monitor a(nother) file, just create a new Monitor, passing it a ScheduledExecutorService. Starting and stopping monitoring is straightforward. You can (should) reuse the same executor for multiple monitors.
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
public interface Event
{
}
public interface Listener
{
void handle(Event event);
}
public class Monitor
{
private static final int CHECK_EVERY_SECONDS = 10;
private static final int RECHECK_AFTER_IF_NOT_EXISTS_SECONDS = 30;
private File file;
private ScheduledExecutorService executor;
private boolean active;
private List<Listener> listeners;
public Monitor(File file, ScheduledExecutorService executor)
{
super();
this.file = file;
this.executor = executor;
listeners = new ArrayList<Listener>();
}
public synchronized void start()
{
if (active)
{
return;
}
active = true;
executor.execute(new Runnable()
{
public void run()
{
synchronized (Monitor.this)
{
if (!active)
{
System.out.println("not active");
return;
}
}
if (!file.exists())
{
System.out.println("does not exist, rescheduled");
executor.schedule(this, RECHECK_AFTER_IF_NOT_EXISTS_SECONDS, TimeUnit.SECONDS);
return;
}
Event event = doStuff(file);
System.out.println("generated " + event);
updateListeners(event);
System.out.println("updated listeners and rescheduled");
executor.schedule(this, CHECK_EVERY_SECONDS, TimeUnit.SECONDS);
}
});
}
private Event doStuff(final File file)
{
return new Event()
{
public String toString()
{
return "event for " + file;
}
};
}
public synchronized void stop()
{
active = false;
}
public void addListener(Listener listener)
{
synchronized (listeners)
{
listeners.add(listener);
}
}
public void removeListener(Listener listener)
{
synchronized (listeners)
{
listeners.remove(listener);
}
}
private void updateListeners(Event event)
{
synchronized (listeners)
{
for (Listener listener : listeners)
{
listener.handle(event);
}
}
}
public static void main(String[] args) throws IOException
{
ScheduledExecutorService executor = Executors.newScheduledThreadPool(4);
File file = new File("test.png");
Monitor monitor = new Monitor(file, executor);
monitor.addListener(new Listener()
{
public void handle(Event event)
{
System.out.println("handling " + event);
}
});
monitor.start();
System.out.println("started...");
System.in.read();
monitor.stop();
System.out.println("done");
executor.shutdown();
}
}
See this post How to start/stop/restart a thread in Java?
I assume you answered your question
one can not restart a dead thread
This link may be helpful to you How to restart thread in java?
A thread in Java cannot be re-started. Every time you need to restart the thread you must make a new one.
That said, you might want to look at:
private void setRandomFile() {
if (!monitoredFile.exists()) {
log.warn("File [" + monitoredFile.getAbsolutePath()
+ "] not exist,will try again after 30 seconds");
try {
Thread.sleep(30 * 1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
setRandomFile();
return;
}
// ....
}
Here you sleep for 30 seconds if the file does not exist, then recursively call the same function. Now, I don't know what business requirements you have, but if this recursion ran long enough you will run out of stack space. Perhaps you will be better served with a while loop or even better, a little synchronisation like a Semaphore.