I was trying to implement some Consumer-Producer problem with BlockingQueue. To do it with some purpose, I decided to write file searching tool.
I decided that search mechanism is working recursively, and every new directory is going to have new thread pool to increase speed of searching.
My problem is, that I have no idea how can I implement mechanism that stops printing threads (consumers) at the end - when searching threads get job done.
I was trying to do that with some ideas like POISON PILLS, but it doesn't works well (threads stops before print any results). Any ideas how can I do that?
Here is some code:
Searching mechanism:
public class SearchingAlgorithm implements Runnable {
private final File file;
private BlockingQueue<File> queue;
private ExecutorService executor;
public SearchingAlgorithm(File fileName, BlockingQueue<File> queue) {
this.file = fileName;
this.queue = queue;
this.executor = Executors.newWorkStealingPool();
}
#Override
public void run() {
try {
searchDeep();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private void searchDeep() throws InterruptedException {
File[] files = file.listFiles();
if (files != null) {
for (File fil : files) {
if (fil.isDirectory()) {
executor.submit(new SearchingAlgorithm(fil, this.queue));
} else {
this.queue.add(fil);
}
}
}
}
}
Printer:
public class ContainingCheckAlgorithm implements Runnable {
private BlockingQueue<File> queue;
// private ExecutorService executor;
private String keyWord;
public ContainingCheckAlgorithm(BlockingQueue<File> queue, String keyWord) {
this.queue = queue;
this.keyWord = keyWord;
// executor = Executors.newFixedThreadPool(2);
}
#Override
public void run() {
try {
printFile();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private void printFile() throws InterruptedException {
while (true) {
File takeFile = queue.take();
String fileName = takeFile.getAbsolutePath()
.toLowerCase();
boolean isContainingKeyWord = fileName.contains(keyWord.toLowerCase());
if (isContainingKeyWord) {
System.out.println(takeFile.getAbsolutePath());
}
}
}
}
Main test class:
public class MainClass {
public static void main(String[] args) throws InterruptedException {
ExecutorService executor = Executors.newFixedThreadPool(2);
BlockingQueue<File> queue = new LinkedBlockingQueue<>();
File fileName = new File("C:/");
SearchingAlgorithm sa = new SearchingAlgorithm(fileName, queue);
executor.submit(sa);
ContainingCheckAlgorithm ca = new ContainingCheckAlgorithm(queue, "Slipknot");
executor.submit(ca);
executor.shutdown();
}
}
Split the whole work in 2 stages. At the first stage, SearchingAlgorithm's work and ContainingCheckAlgorithm waits for new jobs if the queue is empty. At second stage, all SearchingAlgorithm instances finished, and ContainingCheckAlgorithm quits if finds the queue empty. To discover if the queue is empty, ContainingCheckAlgorithm uses queue.poll(timeout) instead of queue.take().
And you need not to create new thread pool for each SearchingAlgorithm.
As u said, I try to do it this way:
Searching algorithm share Thread Pool with others insatances of searchingAlgorithm.
SEARCHING:
public class SearchingAlgorithm implements Runnable {
private final File file;
private BlockingQueue<File> queue;
private ExecutorService executor;
public SearchingAlgorithm(File fileName, BlockingQueue<File> queue, ExecutorService executor) {
this.file = fileName;
this.queue = queue;
this.executor = executor;
}
#Override
public void run() {
try {
searchDeep();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private void searchDeep() throws InterruptedException {
File[] files = file.listFiles();
if (files != null) {
for (File fil : files) {
if (fil.isDirectory()) {
executor.submit(new SearchingAlgorithm(fil, this.queue, executor));
} else {
this.queue.add(fil);
}
}
}
}
Now ContainingCheckAlgorith needs to share CountDownLatch with main class, because I need some mechanism to close Thread Pool in main class. Also it uses pool(timeout) as u said, and my threads finally finishing thier job.
CHECKING
public class ContainingCheckAlgorithm implements Runnable {
private BlockingQueue<File> queue;
private String keyWord;
private CountDownLatch latch;
public ContainingCheckAlgorithm(BlockingQueue<File> queue, String keyWord, CountDownLatch latch) {
this.queue = queue;
this.keyWord = keyWord;
this.latch = latch;
}
#Override
public void run() {
try {
printFile();
latch.countDown();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private void printFile() throws InterruptedException {
File takeFile;
while ((takeFile = queue.poll(1, TimeUnit.SECONDS)) != null) {
String fileName = takeFile.getName()
.toLowerCase();
boolean isContainingKeyWord = fileName.contains(keyWord.toLowerCase());
if (isContainingKeyWord) {
System.out.println(takeFile.getAbsolutePath());
}
}
}
MAIN:
public class MainClass {
public static void main(String[] args) throws InterruptedException {
ExecutorService executor = Executors.newCachedThreadPool();
BlockingQueue<File> queue = new LinkedBlockingQueue<>();
CountDownLatch latch = new CountDownLatch(1);
File fileName = new File("C:/");
SearchingAlgorithm sa = new SearchingAlgorithm(fileName, queue, executor);
executor.submit(sa);
ContainingCheckAlgorithm ca = new ContainingCheckAlgorithm(queue, "Slipknot", latch);
executor.submit(ca);
latch.await();
executor.shutdown();
}
It looks weird, but I wonder what if:
More than 1 thread would run as ContainingCheckAlgorithm ?
SearchingAlgorithm would searching for an file more than 1 second, and ContainingCheckAlgorithm finish work ? Obviously, I can change timeout to 2 second, and more, but we always try to optimize our programs.
Related
In listing 7.15 of "Java Concurrency in Practice" there is an example of producer/consumer with no wait for termination on the method stop.
The main thread in the original code does not wait for the log thread to finish. This can cause issue especially when logging to a file and the file needs to be manipulated after the stop. An exception can be generated if the consumer is still using the file.
Does the following code work adding wait for termination with timeout on the stop method?
public class LogService {
private final BlockingQueue<String> queue;
private final LoggerThread loggerThread;
private final PrintWriter writer;
private final Object lock;
private boolean shutdown;
private boolean prepareShutdown;
private int reservations;
public LogService(Writer writer) {
this.queue = new LinkedBlockingQueue<String>();
this.loggerThread = new LoggerThread();
this.writer = new PrintWriter(writer);
this.lock = new Object();
}
public void start() {
loggerThread.start();
}
public void stop(long timeout) throws InterruptedException {
synchronized (lock) {
prepareShutdown = true;
}
loggerThread.interrupt();
try {
loggerThread.join(timeout);
} finally {
synchronized (lock) {
shutdown = true;
}
loggerThread.interrupt();
}
}
public void log(String msg) throws InterruptedException {
synchronized (lock) {
if (shutdown || prepareShutdown)
throw new IllegalStateException(/* ... */);
++reservations;
}
queue.put(msg);
}
private class LoggerThread extends Thread {
public void run() {
try {
while (true) {
try {
synchronized (lock) {
if (shutdown || prepareShutdown
&& reservations == 0)
break;
}
String msg = queue.take();
synchronized (lock) {
--reservations;
}
writer.println(msg);
} catch (InterruptedException e) { /* retry */
}
}
} finally {
writer.close();
}
}
}
}
I'm currently working on java application which has a scenario of multiple producers adding tasks to a queue and whenever queue is not empty tasks should be executed at predefined rate. (using multiple threads to maintain execution rate) After executing the available tasks executor has to wait till tasks available in the queue again.
I know blockingQueue can be used to triggering part in here and ScheduledExecutorService for execute tasks at fixed rate. But I could not find a way to link ability of both of this for my need. So I would be very thankful if you could give me any suggestion to make this happen.
You need the task queue to be accessible by both the producer and consumer threads. I've written a basic program to demonstrate this, but I'll let you play around with the BlockingQueue API and the ScheduledExecutor as per your needs:
import java.util.concurrent.*;
public class ProducerConsumer {
private static final BlockingQueue<Integer> taskQueue = new LinkedBlockingQueue<>();
public static void main(String[] args) {
ExecutorService consumers = Executors.newFixedThreadPool(3);
consumers.submit(new Consumer());
consumers.submit(new Consumer());
consumers.submit(new Consumer());
ExecutorService producers = Executors.newFixedThreadPool(2);
producers.submit(new Producer(1));
producers.submit(new Producer(2));
}
private static class Producer implements Runnable {
private final int task;
Producer(int task) {
this.task = task;
}
#Override
public void run() {
System.out.println("Adding task: " + task);
taskQueue.add(task); // put is better, since it will block if queue is full
}
}
private static class Consumer implements Runnable {
#Override
public void run() {
try {
Integer task = taskQueue.take(); // block if there is no task available
System.out.println("Executing task: " + task);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
This is the way I could come up with as a solution. It looks little bit rusty but I have tested this and the code is working.
package test;
import java.util.concurrent.*;
public class FixedRateConsumer {
private BlockingQueue<String> queue = new ArrayBlockingQueue<>(20);
private ScheduledExecutorService executorService = new ScheduledThreadPoolExecutor(5);
private boolean continueRunning = true;
public void executeInBackGraound() throws InterruptedException, ExecutionException {
while (continueRunning) {
String s = queue.take();
Worker w = new Worker(s);
ScheduledFuture future = executorService.scheduleAtFixedRate(w, 0, 1, TimeUnit.SECONDS);
w.future = future;
try {
if (!future.isDone()) {
future.get();
}
} catch (CancellationException e) {
// Skipping
}
}
}
public void setContinueRunning(boolean state) {
continueRunning = state;
}
public void addConsumableObject(String s) throws InterruptedException {
queue.put(s);
}
private void consumeString(String s) {
System.out.println("Consumed -> " + s + ", ... # -> " + System.currentTimeMillis() + " ms");
}
private class Worker implements Runnable {
String consumableObject;
ScheduledFuture future;
public Worker(String initialConsumableObject) {
this.consumableObject = initialConsumableObject;
}
#Override
public void run() {
try {
if (consumableObject == null) {
consumableObject = queue.take();
}
consumeString(consumableObject);
consumableObject = null;
if (queue.isEmpty()) {
if (future == null) {
while (future == null) {
Thread.sleep(50);
}
}
future.cancel(false);
}
} catch (Exception e) {
System.out.println("Exception : " + e);
}
}
}
}
I am trying to implement a small producer-consumer example in Java using ExecutorService.
Here is my main class
class Example {
public static void main(String args[]) {
BlockingQueue<String> queue = new ArrayBlockingQueue<>(1000);
Producer producer = new Producer(queue);
Consumer consumer = new Consumer(queue);
ExecutorService executor = Executors.newCachedThreadPool();
// executor.execute(consumer);
Future producerFuture = executor.submit(producer);
Future consumerFuture = executor.submit(consumer);
try {
producerFuture.get();
consumerFuture.get();
} catch (InterruptedException e) {
LOG.error("Failed");
}
executor.shutdown();
executor.awaitTermination(10, TimeUnit.MILLISECONDS);
}
}
Producer Class
public class Producer implements Runnable {
private BlockingQueue<String> queue;
public Producer(BlockingQueue<String> queue) {
this.queue = queue;
}
#Override
public void run() {
for (int i = 0; i < 10; i++) {
try {
queue.put(i + "HELPPPPP");
} catch (InterruptedException ex) {
Logger.getLogger(MigrationToolProducer.class.getName()).log(Level.SEVERE, null, ex);
}
}
Consumer Class
public class Consumer implements Runnable {
private final BlockingQueue<String> queue;
private volatile boolean keepRunning = true;
public Consumer(BlockingQueue<String> queue) {
this.queue = queue;
}
#Override
public void run() {
while (keepRunning) {
String value;
try {
value = queue.take();
} catch(InterruptedException e) {
throw new RuntimeException(e);
}
System.out.println(value);
}
}
}
EDIT The execution is stuck at queue.take() in Consumer Class.
Can anyone please help me fix this problem ? Why is the execution stuck in the consumer ?
One possible solution:
1) On Producer side, put a "END" signal after original 10 puts:
queue.put("END");
2) On Consumer side, once detect "END" signal, break the loop:
public void run() {
while (keepRunning) {
String value;
try {
value = queue.take();
if(value.equals("END")) {
System.out.println("Get END signal. All done!");
break;
}
} catch(InterruptedException e) {
throw new RuntimeException(e);
}
System.out.println(value);
}
}
I have a worker threadpool set up that executes a bit of work which I want to log in a central place.
To be more precise, I've extended the Thread class into a worker class, which checks the status of a concurrent queue. If it's empty, then it waits. As elements are added by another thread, notify() wakes the workers. Once they've completed the task, they wait for the next element in the queue.
What's the best practice to have each of the threads report their status at the end of each of their tasks?
public class PoolWorker extends Thread {
public ConcurrentLinkedQueue<Device> q;
public PoolWorker(ConcurrentLinkedQueue<Device> q, String type){
this.q = q;
this.type = type;
}
#Override
public void run(){
while (true)
{
Device d = null;
try{
synchronized(q){
while(q.isEmpty())
{
q.wait(); // wait for a notify()
}
d = q.remove();
}
// do some work
// report status of work completed
}
}
Try to do something like this
ExecutorService exec = Executors.newFixedThreadPool(10);
Runnable runn = new Runnable()
{
#Override
public void run()
{
System.out.println("");
}
};
exec.execute(runn);
As mentioned best way is to use BlockingQueue. Below is the sample code:
public class PoolWorker extends Thread {
public ArrayBlockingQueue<String> q;
public String type;
public PoolWorker(ArrayBlockingQueue<String> q, String type) {
this.q = q;
this.type = type;
}
#Override
public void run() {
while(true){
String work = null;
try {
System.out.println("PoolWorker.run:waiting .............");
work = q.take();
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("PoolWorker.run..work: " + work);
}
}
public static void main(String[] args) throws InterruptedException {
ArrayBlockingQueue<String> pool = new ArrayBlockingQueue<String>(100);
PoolWorker worker = new PoolWorker(pool, "Something");
worker.start();
addWork(pool, "work1");
addWork(pool, "work2");
addWork(pool, "work3");
addWork(pool, "work4");
addWork(pool, "work5");
//Just give enough time to run
Thread.sleep(5000);
}
private static void addWork(ArrayBlockingQueue<String> pool, String work) throws InterruptedException {
System.out.println("PoolWorker.addWork: " + work);
pool.put(work);
}
}
There is nice sample code available in Java documentation as well:
http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/BlockingQueue.html
I have a class that iterates through a list of links. For each link I want to do a treatment. So I have to create a thread for each link.
Here is the code (Main):
ThreadProcessing tp;
for(int i = 0; i < listUrl.size(); i++)
{
tp.add(string_url);
}
For the ThreadProcessing class, I have to use the Executor interface.
The point is : I have to create a pool of 30 threads. ThreadP class contains a list of non-blocking threads (it can contain more than 30 threads of course). You can add as many threads as you want and the class will be responsible to perform all these threads.
So that's what I tried to do (it does not work).The ThreadProcessing class :
public class ThreadProcessing {
List<Runnable> runnables = new ArrayList<Runnable>();
ExecutorService pool;
#PostConstruct
public void init()
{
pool = Executors.newFixedThreadPool(30);
}
public void add(String url)
{
runnables.add(createRunnable(url));
executeRunnables(pool, runnables);
}
public static void executeRunnables(final ExecutorService service, List<Runnable> runnables){
for(Runnable r : runnables){
service.execute(r);
}
service.shutdown();
}
private Runnable createRunnable(final String url){
Runnable getContentFromURL = new Runnable(){
public void run(){
//My treatment with url
}
};
return getContentFromURL;
}
}
I hope I have not been too vague in my explanation, thank you.
public void add( String url) {
Runnable job = createRunnable(url);
runnables.add( job);
pool.execute( job);
}
Also, do not shut the pool down unless you are finished submitting/adding jobs. Of course, in this example, you don't really need the runnables List.
Try something like:
public void main() {
ExecutorService es = Executors.newFixedThreadPool(30);
BlockingQueue<String> urls =
new ArrayBlockingQueue<String>(listUrl.size(), false, listUrl);
LinkedList<Future<?>> futures = new LinkedList<Future<?>>();
for(int i = 0 ; i < 30 ; ++i) {
futures.add(es.submit(new URLRunnable(urls)));
}
// Wait for all the futures to return
for(Future<?> f : futures) {
f.get();
}
}
public class URLRunnable() implements Runnable() {
private final BlockingQueue<String> urls;
URLRunnable(BlockingQueue<String> urls) { this.urls = urls; }
#Override
public void run() {
String url = null;
while((url = urls.poll()) != null) {
// do something with url
}
}
}