Develop Reference

Call method after each cycle of ScheduledExecutorService

I need the following scenario:
Run all ScheduledFutures within a cycle and call every time the method tasksCompleted() after all tasks finished its execution. The next scheduling cycle must not wait while calling tasksCompleted() after the actual scheduling cycle.
In short: Call a method after the completion of the actual scheduling-cycle and do not stop the next scheduling-cycle
The following code creates tasks and the scheduling works. However, I am not able call tasksCompleted() when all tasks within a cycle completed.
import java.time.LocalTime;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;
public class Scheduler {
public static void main(String[] args) {
final ScheduledExecutorService ses = Executors.newScheduledThreadPool(10);
System.out.println("- [" + LocalTime.now() + "] run parent-task...");
// create 3 tasks: each task needs 7 seconds.
var tasks = createTasks("child", 3, 7);
List<ScheduledFuture<?>> futures = new ArrayList<>();
tasks.forEach(t ->
{
ScheduledFuture<?> future = ses.scheduleWithFixedDelay(t, 0, 2, TimeUnit.SECONDS);
futures.add(future);
});
// this does not work..
var scheduleCycleCompleted = futures.stream().allMatch(f -> f.isDone());
System.out.println("scheduleCycleCompleted: " + scheduleCycleCompleted);
// maybe a solution with CompletableFuture?
CompletableFuture[] cfs = futures.toArray(new CompletableFuture[futures.size()]);
}
static void tasksCompleted() {
System.out.println("schedule cycle completed");
}
static List<Runnable> createTasks(String group, int numbersOfTasks, long taskDuration) {
var tasks = new ArrayList<Runnable>();
for (var i = 0; i < numbersOfTasks; i++) {
int taskNr = i;
Runnable task = () ->
{
System.out.println("- [" + LocalTime.now() + "] Running " + group + "-task" + taskNr + "...[needs "
+ taskDuration + " seconds]");
try {
TimeUnit.SECONDS.sleep(taskDuration);
} catch (InterruptedException e) {
e.printStackTrace();
Thread.currentThread().interrupt();
}
};
tasks.add(task);
}
return tasks;
}
}

Updated
I hope it will work.
CountDownLatch will solve the problem here.
import java.time.LocalTime;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.*;
public class Scheduler {
public static void main(String[] args) throws InterruptedException {
final ScheduledExecutorService ses = Executors.newScheduledThreadPool(10);
System.out.println("- [" + LocalTime.now() + "] run parent-task...");
int noOfTask=3;
CountDownLatch countDownLatch = new CountDownLatch(noOfTask);
TaskComplete taskComplete=new TaskCompleteImpl(noOfTask,countDownLatch);
// create 3 tasks: each task needs 7 seconds.
List<Runnable> tasks = createTasks("child", noOfTask, 7,countDownLatch,taskComplete);
List<ScheduledFuture<?>> futures = new ArrayList<>();
tasks.forEach(t ->
{
ScheduledFuture<?> future = ses.scheduleWithFixedDelay(t, 0, 2, TimeUnit.SECONDS);
futures.add(future);
});
// this does not work..
}
interface TaskComplete{
void tasksCompleted();
}
static class TaskCompleteImpl implements TaskComplete {
int totalTask=0;
int index=0;
CountDownLatch countDownLatch;
public TaskCompleteImpl(int totalTask){
}
public TaskCompleteImpl(int noOfTask, CountDownLatch countDownLatch) {
this.totalTask=noOfTask;
this.countDownLatch=countDownLatch;
}
#Override
public synchronized void tasksCompleted() {
index=index+1;
if(index==totalTask){
System.out.println("schedule cycle completed");
index=0;
countDownLatch=new CountDownLatch(totalTask);
}
}
}
static List<Runnable> createTasks(String group, int numbersOfTasks, long taskDuration, CountDownLatch countDownLatch, TaskComplete taskComplete) {
List tasks = new ArrayList<Runnable>();
for (int i = 0; i < numbersOfTasks; i++) {
int taskNr = i;
Runnable task = () ->
{
System.out.println("- [" + LocalTime.now() + "] Running " + group + "-task" + taskNr + "...[needs "
+ taskDuration + " seconds]");
try {
TimeUnit.SECONDS.sleep(taskDuration);
countDownLatch.countDown();
countDownLatch.await();
taskComplete.tasksCompleted();
} catch (InterruptedException e) {
e.printStackTrace();
Thread.currentThread().interrupt();
}
};
tasks.add(task);
}
return tasks;
}
}

To handle the case of different runtimes within a scheduling cycle, what you need is a way to identify which task belongs to which group. You can do that by giving them an identifying counter, so that each time a task is executed, the counter is used to denominate the group it's running in.
interface GroupedRunnable extends Runnable {
String getGroup();
}
class CountingRunnable implements GroupedRunnable {
private AtomicInteger counter = new AtomicInteger();
private final Runnable delegate;
private final String taskName;
CountingRunnable(Runnable delegate, String taskName) {
this.delegate = delegate;
this.taskName = taskName;
}
public void run() {
System.out.printf("[%s] - Running task %s in group %s%n", LocalTime.now(), taskName, getGroup());
delegate.run();
counter.incrementAndGet();
System.out.printf("[%s] - Running task %s in group %s finished%n", LocalTime.now(), taskName, getGroup());
}
#Override
public String getGroup() {
return counter.toString();
}
}
Now, you can have a watchdog class that keeps track on which members of which group have already executed. Since you know how many members a group has, a simple counter is sufficient.
class GroupMonitoringService {
// key: group, value: tasks
Map<String, AtomicInteger> finishedTasks = new HashMap<>();
private final Runnable finisher;
private final int groupSize;
GroupMonitoringService(Runnable finisher, int groupSize) {
this.finisher = finisher;
this.groupSize = groupSize;
}
public synchronized void taskFinished(String group) {
var finishedInGroup = finishedTasks.computeIfAbsent(group, k -> new AtomicInteger());
if (finishedInGroup.incrementAndGet() >= groupSize) {
// scheduling group complete
System.out.printf("Group %s finished executing%n", group);
finisher.run();
finishedTasks.remove(group);
}
}
}
Now all you have to do is wrap your original task into a grouped task, and make sure the monitoring service is notified when it's finished, so wrap again.
private static List<Runnable> createTasks() {
List<Runnable> result = new ArrayList<>();
for (int i = 0; i < GROUP_SIZE; i++) {
RandomWaitTask originalTask = new RandomWaitTask();
CountingRunnable groupedTask = new CountingRunnable(originalTask, "Task " + i);
Runnable notifyingRunnable = () -> {
groupedTask.run();
MONITORING_SERVICE.taskFinished(groupedTask.getGroup());
};
result.add(notifyingRunnable);
}
return result;
}
So now you can just schedule those.
You can see the entire code here (although it doesn't actually run properly on that site because of the resource limit it imposes, but if you copy it into your IDE, it works).

Calculating time taken by my worker threads when it is still processing

I am not able to get the processing times for my worker threads. I have a thread monitoring the processing times for my worker threads and reports the time taken and how many of the worker threads are in processing state.
Questions
My monitoring thread, always reports that there are no workers in processing state? I suspect my monitoring thread is not able to evaluate the worker threads appropriately.
I have synchronised some of the methods that fetch and record the processing times and processing state. Is that good enough for my monitor thread to keep track of processing times without thread overwrites.
Should the isProcessing boolean variable be a volatile variable?
MyWorker class
public class MyWorker implements Runnable {
private final int WAIT_INTERVAL = 200;
private MyService myService;
private MyProvider myProvider;
private boolean stopRequested = false;
private boolean isProcessing; // Flag to indicate the worker is currently processing a message
private long processingStartTime; //captures when the worker started processing a message
private Logger logger = new Logger();
public MyWorker(MyService myService, MyProvider myProvider){
this.myService = myService;
this.myProvider = myProvider;
this.isProcessing = false;
this.processingStartTime = -1;
}
public void setStopRequested() {
stopRequested = true;
}
private synchronized void recordProcessingStart(long start){
isProcessing = true;
processingStartTime = start;
}
private synchronized void recordProcessingStop(){
isProcessing = false;
processingStartTime = -1;
}
public synchronized boolean isProcessing(){
return isProcessing;
}
public synchronized long getProcessingStartTime(){
return processingStartTime;
}
#Override
public void run() {
while (!stopRequested) {
boolean processingMessage = false;
List<Message> messages = myProvider.getPendingMessages();
if (messages.size() != 0) {
logger.log("We have " + messages.size() + " messages");
recordProcessingStart(System.currentTimeMillis());
for (Message message : messages) {
processMessage(messages);
}
recordProcessingStop();
}
if (!(processingMessage || stopRequested)) {
// this is to stop the thread from spinning when there are no messages
try {
Thread.sleep(WAIT_INTERVAL);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
private void processMessage(Message messages){
myService.process(messages);
}
}
WorkerManager class
public class WorkerManager implements Runnable {
private MyWorker[] workers;
private int workerCount;
private boolean stopRequested;
private ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor();
private Logger logger = new Logger();
public WorkerManager(int count) {
this.workerCount = count;
}
#Override
public void run() {
stopRequested = false;
boolean managerStarted = false;
while (!stopRequested) {
if (!managerStarted) {
workers = new MyWorker[workerCount];
for (int i = 0; i < workerCount; i++) {
final Thread workerThread = new Thread(workers[i], "Worker-" + (i + 1));
workerThread.start();
}
startProcessMonitoringThread();
managerStarted = true;
}
}
}
public void stop() {
stopRequested = true;
}
public void cleanUpOnExit() {
for (MyWorker w : workers) {
w.setStopRequested();
}
stopProcessMonitoringThread();
}
private void startProcessMonitoringThread(){
scheduledExecutorService = Executors.newScheduledThreadPool(1);
scheduledExecutorService.scheduleWithFixedDelay( new WorkerMonitorThread(workers), 1, 1, TimeUnit.MINUTES);
logger.info("Monitoring of worker threads initialized ...");
}
private void stopProcessMonitoringThread(){
scheduledExecutorService.shutdown();
logger.info("Successfully shutdown worker monitoring thread ...");
}
private class WorkerMonitorThread implements Runnable{
private MyWorker[] workers;
WorkerMonitorThread(MyWorker workers[]){
this.workers = workers;
}
#Override
public void run() {
String trackingId = "["+ UUID.randomUUID().toString() + "] - ";
logger.info(trackingId + "Calculating processing times of worker threads ...");
StringBuilder sb = new StringBuilder();
int totalWorkersInProcess = 0;
for (int i = 0; i < workerCount; i++) {
MyWorker worker = workers[i];
if(worker.isProcessing()){
totalWorkersInProcess++;
long startTime = worker.getProcessingStartTime();
long currentTime = System.currentTimeMillis();
long duration = currentTime - startTime;
sb.append(trackingId + "Worker-" + (i + 1) + " has been processing for the last : " + duration + " ms");
}
}
sb.append(trackingId + "Total worker(s) in progress : " + totalWorkersInProcess);
logger.info(sb.toString());
}
}
}

You haven´t created your workers.
Review that code of your WorkerManager run method:
workers = new MyWorker[workerCount]; // Create an array of MyWorker, but each element of the array is null.
for (int i = 0; i < workerCount; i++) {
final Thread workerThread = new Thread(workers[i], "Worker-" + (i + 1)); // <-- Creates a thread and pass a null MyWorker.
workerThread.start(); // NullPointers are coming on the other thread.
}
To fix the bug your WorkerManager should create the workers or receive the workers to use. You could add to the constructor the dependencies to create the workers:
public WorkerManager(int count, MyService myService, MyProvider myProvider) {
this.workerCount = count;
this.myService = myService;
this.myProvider = myProvider;
}
And then, create the workers correctly:
workers = new MyWorker[workerCount];
for (int i = 0; i < workerCount; i++) {
workers[i] = new MyWorker(myService, myProvider);
}

why the maxPoolSize in the ThreadPoolExecutor do nothing? [duplicate]

This question already has answers here:
How to get the ThreadPoolExecutor to increase threads to max before queueing?
(10 answers)
Closed 5 years ago.
In the ThreadPoolExecutor class, there is a maxPoolSize to specify the maximum thread pool size. That means if number of threads less than that number, threads in the pool should be executed immediately. But I found it is not the case. It cannot go beyond the corePoolSize actually. I am confused. what is the purpose of the maxPoolSize if it does nothing? Here is my testing program:
I have specified the corePoolSize = 2; maxPoolSize = 6; I have created 5 threads (Runnable). I thought all the 5 threads (Runnable) should running simultaneously. But they are not. Only two of them are running, another 3 are put aside until the first two die.
I have read many posts about the topic. But none can guide me to make the 5 threads running simultaneously.
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
public class MyThreadPoolExecutorTest2
{
private List<MyRunnable> myRunnables = new ArrayList<>();
public static void main(String[] args)
{
new MyThreadPoolExecutorTest2().test();
}
public void test()
{
int poolSize = 2;
int maxPoolSize = 6;
int threadPoolKeepAliveTimeInSec = 30;
ExecutorService threadPoolExecutor =
new MySimpleThreadPoolExecutor(poolSize, maxPoolSize, threadPoolKeepAliveTimeInSec);
int numOfThread = 5;
System.out.println("Start thread pool test with corePoolSize=" + poolSize + ", maxPoolSize=" + maxPoolSize
+ ", actualThreads=" + numOfThread);
for (int i = 0; i < numOfThread; i++)
{
MyRunnable tempRunnable = new MyRunnable(i + 1, "PoolTest" + (i + 1));
myRunnables.add(tempRunnable);
threadPoolExecutor.execute(tempRunnable);
}
System.out.println("********* wait for a while");
try
{
Thread.sleep(20000);
}
catch (InterruptedException e)
{
e.printStackTrace();
}
System.out.println("^^^^^^^^^^ shutdown them all");
for (MyRunnable runnable : myRunnables)
{
runnable.shutdown();
}
System.out.println("Ended thread pool test.");
}
public class MyRunnable implements Runnable
{
private int id = 0;
private String name = "";
private boolean shutdown = false;
public MyRunnable(int id, String name)
{
this.id = id;
this.name = name;
}
#Override
public void run()
{
System.out.println("++++ Starting Thread: " + id + ":" + name);
while (!shutdown)
{
try
{
Thread.sleep(200);
}
catch (InterruptedException e)
{
e.printStackTrace();
}
}
System.out.println("---- Ended Thread: " + id + ":" + name);
}
public void shutdown()
{
shutdown = true;
}
}
}
class MySimpleThreadPoolExecutor extends ThreadPoolExecutor
{
private static int peakActiveThreads = 0;
private String taskInfo = "";
public MySimpleThreadPoolExecutor(int nThreads, int maxThreads, int threadPoolKeepAliveTimeInSec)
{
super(nThreads, maxThreads, threadPoolKeepAliveTimeInSec * 1000L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>());
System.out.println("MySimpleThreadPoolExecutor::MySimpleThreadPoolExecutor(), threadPoolSize=" + nThreads
+ ", maxThreadCount=" + maxThreads + ", threadPoolKeepAliveTimeInSec=" + threadPoolKeepAliveTimeInSec);
}
#Override
public void beforeExecute(Thread t, Runnable r)
{
int activeCount = getActiveCount();
if (MySimpleThreadPoolExecutor.peakActiveThreads < activeCount)
{
MySimpleThreadPoolExecutor.peakActiveThreads = activeCount;
}
taskInfo = r.toString();
String msg =
"BeforeE thread(name:id)::" + t.getName() + ":" + t.getId() + ", task::" + r.toString() + "\n"
+ threadPoolInfoStr();
System.out.println("ThreadInfo before, MySimpleThreadPoolExecutor::beforeExecute(), " + msg);
super.beforeExecute(t, r);
}
#Override
public void execute(Runnable command)
{
beforeExecute(Thread.currentThread(), command);
super.execute(command);
}
public String threadPoolInfoStr()
{
return String.format("Thead: %s/%d\n[PoolSize/CorePoolSize] [%d/%d]\nActive: %d\nCompleted: %d\nTask: %d"
+ "\nisShutdown: %s\nisTerminated: %s\npeakActiveThreads: %d\nTaskInfo: %s\nQueueSize: %d", Thread
.currentThread().getName(), Thread.currentThread().getId(), getPoolSize(), getCorePoolSize(),
getActiveCount(), getCompletedTaskCount(), getTaskCount(), isShutdown(), isTerminated(),
MySimpleThreadPoolExecutor.peakActiveThreads, taskInfo, getQueue().size());
}
}

New threads will only be created up to maxPoolSize once the queue is full. Before, the limit is the one defined at corePoolSize.
Reference: http://www.bigsoft.co.uk/blog/index.php/2009/11/27/rules-of-a-threadpoolexecutor-pool-size

More threads are added only if the queue is full.
Since your LinkedBlockingQueue is not bounded, it will never be full. Therefore, there will never be more than the core pool size of threads in the pool.
Use a TransferQueue or use a bounded queue to fix this.

Why is ExecutorService.awaitTermination() succeeding before a submitted task has completed

In my code I have a number of executorservices that run as a pipeline in that the first executorService may submit a task to any later executor service but never the other way round.
services.add(songLoaderService);
services.add(AcoustIdMatcher.getExecutorService());
services.add(SongPrematcherMatcher.getExecutorService());
services.add(MusicBrainzMetadataMatcher.getExecutorService());
//Start Loading Songs
songLoaderService.submit(loader);
We only submit one task for first service then I can request a shutdown. This will not succeed until this task has completed and by then it will have put some tasks on to the second service and so on.
So this code has always worked for a number of years, shutdown() is never called until all tasks that were going to be submitted have been, and the awaitTermination() method does not until all the tasks that were submitted have completed.
int count = 0;
for (ExecutorService service : services)
{
MainWindow.logger.severe("Requested Shutdown Task:" + count + ":"+((SongKongThreadFactory)((TimeoutThreadPoolExecutor) service).getThreadFactory()).getName());
//Request Shutdown
service.shutdown();
//Now wait for service to terminate
service.awaitTermination(10, TimeUnit.DAYS);
MainWindow.logger.severe("Completed Shutdown Task:" + count);
if(count==2)
{
MainWindow.logger.severe("Report:"+currentReportId+":SongPreMatcher:" + SongPrematcherMatcher.getPipelineQueuedCount()+":"+ SongPrematcherMatcher.getPipelineCallCount()+":"+ SongPrematcherMatcher.getPipelineCompletedCount()+":"+SongPrematcherMatcher.getPipelineFileCount());
}
count++;
}
But I am now seeing a problem with one ExecutorService not working in this way.
The request to shutdown SongPrematcherMatcher service is succeeding after all tasks added to this service by the previous (AcoustIdMatcher) service have been submitted and started but before one of them has completed, as can be seen by the following debug line
Report:353:SongPreMatcher:init:57:started:57:Finished:56
The missing task hasnt failed since we can see it completes at the end of the log output, but the point is it completes after the service it runs on has been successfully terminated.
This has significant consequences since it means all the tasks that this task trie to submit to the MusicBrainzMetadataMatcher service fail since a shutdown request has already been made for that since the previous service (PrematcherMatched) has been shutdown .
PrematcherMatcher was added quite recently so my assumption is that there is something wrong with it but I cannot see what it can possibly be.
toplevelanalyzer.FixSongsController:start:SEVERE: Requested Shutdown Task:0
analyser.AcoustIdMatcher:<init>:SEVERE: GROUP 115:C:\Users\Paul\Desktop\FlatRandomFolder:true:false:true:false
toplevelanalyzer.FixSongsController:start:SEVERE: Completed Shutdown Task:0
toplevelanalyzer.FixSongsController:start:SEVERE: Requested Shutdown Task:1:analyser.AcoustIdMatcher
analyser.SongPrematcherMatcher:<init>:SEVERE: Queue:GROUP 791:C:\Users\Paul\Desktop\FlatRandomFolder:true:false:true:false
analyser.SongPrematcherMatcher:call:SEVERE: Start:GROUP 791:C:\Users\Paul\Desktop\FlatRandomFolder:true:false:true:false
toplevelanalyzer.FixSongsController:start:SEVERE: Completed Shutdown Task:1
toplevelanalyzer.FixSongsController:start:SEVERE: Requested Shutdown Task:2:analyser.SongPrematcherMatcher
toplevelanalyzer.FixSongsController:start:SEVERE: Completed Shutdown Task:2
toplevelanalyzer.FixSongsController:start:SEVERE: Report:353:SongPreMatcher:init:57:started:57:Finished:56
toplevelanalyzer.FixSongsController:start:SEVERE: Requested Shutdown Task:3:analyser.MusicBrainzMetadataMatcher
analyser.MusicBrainzMetadataMatcher:<init>:SEVERE: Queue:GROUP 795:C:\Users\Paul\Desktop\FlatRandomFolder:true:false:true:false
analyser.MusicBrainzMetadataMatcher:<init>:SEVERE: Queue:GROUP 797:C:\Users\Paul\Desktop\FlatRandomFolder:true:false:true:false
analyser.MusicBrainzMetadataMatcher:<init>:SEVERE: Queue:GROUP 799:C:\Users\Paul\Desktop\FlatRandomFolder:true:false:true:false
analyser.MusicBrainzMetadataMatcher:<init>:SEVERE: Queue:GROUP 821:C:\Users\Paul\Desktop\FlatRandomFolder:true:false:true:false
analyser.MusicBrainzMetadataMatcher:<init>:SEVERE: Queue:GROUP 823:C:\Users\Paul\Desktop\FlatRandomFolder:true:false:true:false
analyser.SongPrematcherMatcher:call:SEVERE: Finish:GROUP 791:C:\Users\Paul\Desktop\FlatRandomFolder:true:false:true:false
If I artificially add a delay in the loop then at least for this particular test case it works, but this is not a solution as it introduces delays when there are series of executor services where it would be valid to shutdown them now one after another. Its also not clear why this fix works, and if it would always work.
for (ExecutorService service : services)
{
Thread.sleep(5000);
//Request Shutdown
service.shutdown();
......
}
There is one executorservice for each task, only one type of task can be added to a particular executorservice. The executorService does have special handling to allow user to cancel tasks and to protect against long runing tasks but this was not the issue here.
I cannot see anything different in the PreMatcherMatcher code to any other of the tasks.
package com.jthink.songkong.analyse.analyser;
import com.jthink.songkong.analyse.general.Errors;
import com.jthink.songkong.cmdline.SongKong;
import com.jthink.songkong.ui.MainWindow;
import com.jthink.songkong.util.SongKongThreadFactory;
import java.util.List;
import java.util.concurrent.*;
/**
* From http://stackoverflow.com/questions/2758612/executorservice-that-interrupts-tasks-after-a-timeout
* With additional support for caller running task when bounded queue is full
*/
public class TimeoutThreadPoolExecutor extends ThreadPoolExecutor {
private final long timeout;
private final TimeUnit timeoutUnit;
private final ScheduledExecutorService timeoutExecutor = Executors.newSingleThreadScheduledExecutor();
private final ConcurrentMap<Runnable, ScheduledFuture> runningTasks = new ConcurrentHashMap<Runnable, ScheduledFuture>();
private final static int WAIT_BEFORE_STOP = 10000;
public long getTimeout()
{
return timeout;
}
public TimeUnit getTimeoutUnit()
{
return timeoutUnit;
}
public TimeoutThreadPoolExecutor(int workerSize, ThreadFactory threadFactory, LinkedBlockingQueue<Runnable> queue,long timeout, TimeUnit timeoutUnit)
{
super(workerSize, workerSize, 0L, TimeUnit.MILLISECONDS, queue, threadFactory, new ThreadPoolExecutor.CallerRunsPolicy());
this.timeout = timeout;
this.timeoutUnit = timeoutUnit;
}
#Override
public List<Runnable> shutdownNow() {
timeoutExecutor.shutdownNow();
return super.shutdownNow();
}
#Override
public <T> FutureCallable<T> newTaskFor(Callable<T> callable) {
return new FutureCallable<T>(callable);
}
#Override
protected void beforeExecute(Thread t, Runnable r) {
MainWindow.logger.warning("beforeExecute:"+t.getName()+":"+r.toString());
SongKong.checkIn();
if(timeout > 0) {
final ScheduledFuture<?> scheduled = timeoutExecutor.schedule(new TimeoutTask(t,r), timeout, timeoutUnit);
runningTasks.put(r, scheduled);
}
}
#Override
protected void afterExecute(Runnable r, Throwable t) {
MainWindow.logger.warning("afterExecute:"+r.toString());
//AfterExecute will be called after the task has completed, either of its own accord or because it
//took too long and was interrupted by corresponding timeout task
//Remove mapping and cancel timeout task
ScheduledFuture timeoutTask = runningTasks.remove(r);
if(timeoutTask != null) {
timeoutTask.cancel(false);
}
}
#Override
protected void terminated()
{
//All tasks have completed either naturally or via being cancelled by timeout task so close the timeout task
MainWindow.logger.warning("---Terminated:"+((SongKongThreadFactory)getThreadFactory()).getName());
timeoutExecutor.shutdown();
}
class TimeoutTask implements Runnable {
private final Thread thread;
private Callable c;
public TimeoutTask(Thread thread, Runnable c) {
this.thread = thread;
if(c instanceof FutureCallable)
{
this.c = ((FutureCallable) c).getCallable();
}
}
#Override
public void run()
{
String msg = "";
if (c != null)
{
if (c instanceof AcoustIdMatcher)
{
msg = c.getClass() + ":" + ((AcoustIdMatcher) c).getSongGroup().getKey();
}
else if (c instanceof SongPrematcherMatcher)
{
msg = c.getClass() + ":" + ((SongPrematcherMatcher) c).getSongGroup().getKey();
}
else if (c instanceof MusicBrainzSongGroupMatcher)
{
msg = c.getClass() + ":" + ((MusicBrainzSongGroupMatcher) c).getSongGroup().getKey();
}
else if (c instanceof MusicBrainzMetadataMatcher)
{
msg = c.getClass() + ":" + ((MusicBrainzMetadataMatcher) c).getSongGroup().getKey();
}
else if (c instanceof MusicBrainzUpdateSongOnly)
{
msg = c.getClass() + ":" + ((MusicBrainzUpdateSongOnly) c).getSongGroup().getKey();
}
else if (c instanceof DiscogsSongGroupMatcher)
{
msg = c.getClass() + ":" + ((DiscogsSongGroupMatcher) c).getSongGroup().getKey();
}
else if (c instanceof MusicBrainzSongMatcher)
{
msg = c.getClass() + ":" + String.valueOf(((MusicBrainzSongMatcher) c).getSongId());
}
else if (c instanceof SongSaver)
{
msg = c.getClass() + ":" + String.valueOf(((SongSaver) c).getSongId());
}
else
{
msg = c.getClass().getName();
}
}
if (c != null && c instanceof CancelableTask)
{
MainWindow.logger.warning("+++Cancelling " + msg + " task because taking too long");
((CancelableTask) c).setCancelTask(true);
StackTraceElement[] stackTrace = thread.getStackTrace();
Errors.addError("Cancelled " + msg + " because taken too long", stackTrace);
Counters.getErrors().getCounter().incrementAndGet();
if(stackTrace.length>0)
{
boolean isKnownProblem = false;
for(int i=0;i<stackTrace.length;i++)
{
if(
(stackTrace[i].getClassName().contains("CosineSimilarity")) ||
(stackTrace[i].getClassName().contains("com.jthink.songkong.fileloader.FileFilters"))
)
{
isKnownProblem=true;
break;
}
}
if(isKnownProblem)
{
MainWindow.logger.warning("+++Interrupting " + msg + " task because taking too long");
thread.interrupt();
try
{
Thread.sleep(WAIT_BEFORE_STOP);
}
catch (InterruptedException ie)
{
MainWindow.logger.warning("+++Interrupted TimeoutTask " + msg + " task because taking too long");
}
if(thread.isAlive())
{
MainWindow.logger.warning("+++Stopping CosineSimailarity task");
thread.stop();
}
}
}
}
}
}
}
public class AnalyserService
{
protected static final int BOUNDED_QUEUE_SIZE = 500;
protected String threadGroup;
public AnalyserService(String threadGroup)
{
this.threadGroup=threadGroup;
}
protected ExecutorService executorService;
protected void initExecutorService()
{
int workerSize = Runtime.getRuntime().availableProcessors();
executorService = new PausableExecutor(workerSize, workerSize,0L, TimeUnit.MILLISECONDS,new LinkedBlockingQueue<Runnable>(BOUNDED_QUEUE_SIZE),new SongKongThreadFactory(threadGroup));
}
public ExecutorService getExecutorService()
{
if (executorService == null || executorService.isShutdown())
{
initExecutorService();
}
return executorService;
}
/** Submit and return immediately
*
* #param task
*/
public void submit(Callable<Boolean> task) //throws Exception
{
executorService.submit(task);
}
}
public class AnalyserServiceWithTimeout extends AnalyserService
{
private static final int TIMEOUT_PER_TASK = 30;
public AnalyserServiceWithTimeout(String threadGroup)
{
super(threadGroup);
}
#Override
protected void initExecutorService()
{
int workerSize = Runtime.getRuntime().availableProcessors();
executorService = new TimeoutThreadPoolExecutor(workerSize,
new SongKongThreadFactory(threadGroup),
new LinkedBlockingQueue<Runnable>(BOUNDED_QUEUE_SIZE),
TIMEOUT_PER_TASK,
TimeUnit.MINUTES);
}
}
package com.jthink.songkong.analyse.analyser;
import com.google.common.base.Strings;
import com.jthink.songkong.analyse.general.Errors;
import com.jthink.songkong.cmdline.SongKong;
import com.jthink.songkong.db.SongCache;
import com.jthink.songkong.match.MetadataGatherer;
import com.jthink.songkong.preferences.UserPreferences;
import com.jthink.songkong.ui.MainWindow;
import com.jthink.songkong.util.SongKongThreadGroup;
import com.jthink.songlayer.Song;
import com.jthink.songlayer.hibernate.HibernateUtil;
import org.hibernate.Session;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.logging.Level;
/**
* Try and match songs to acoustid only first as a starting point
*
* Use when we have no or little metadata
*/
public class SongPrematcherMatcher extends CancelableTask implements Callable<Boolean> {
private static PipelineCount pipelineCount = new PipelineCount();
public static int getPipelineQueuedCount()
{
return pipelineCount.getQueuedCount();
}
public static int getPipelineCallCount()
{
return pipelineCount.getCallCount();
}
public static void resetPipelineCount()
{
pipelineCount.resetCounts();
}
public static int getPipelineFileCount()
{
return pipelineCount.getFileCount();
}
public static int getPipelineCompletedCount()
{
return pipelineCount.getCompletedCount();
}
private static AnalyserService analyserService = new AnalyserServiceWithTimeout(SongKongThreadGroup.THREAD_PREMATCHER_WORKER);
private Session session;
private SongGroup songGroup;
public SongGroup getSongGroup()
{
return songGroup;
}
public SongPrematcherMatcher(SongGroup songGroup)
{
SongKong.logger.severe("Queue:"+ songGroup.getKey());
pipelineCount.incQueuedCount();
pipelineCount.incFileCount(songGroup.getSongIds().size());
this.songGroup = songGroup;
}
public static ExecutorService getExecutorService()
{
return analyserService.getExecutorService();
}
public static AnalyserService getService()
{
return analyserService;
}
public Boolean call()
{
try
{
SongKong.logger.severe("Start:" + songGroup.getKey());
if (SongKong.isStopTask() || isCancelTask())
{
return false;
}
SongKong.checkIn();
pipelineCount.incCallCount();
session = HibernateUtil.beginTransaction();
AnalysisStats stats = new AnalysisStats();
List<Song> songs = SongCache.loadSongsFromDatabase(session, songGroup.getSongIds());
//Try to match acoustid this should allow more to be grouped and matched by metadata on first pass
try
{
new RecordingOnlyMatcher().matchRecordingsOnlyByAcoustid(session, songGroup, songs, stats);
}
catch(Exception ex)
{
MainWindow.logger.log(Level.SEVERE, Strings.nullToEmpty(ex.getMessage()), ex);
Errors.addError(Strings.nullToEmpty(ex.getMessage()));
}
session.getTransaction().commit();
HibernateUtil.closeSession(session);
processSongsWithNewMetadata(songGroup, songs);
pipelineCount.incCompletedCount();
SongKong.logger.severe("Finish:" + songGroup.getKey());
return true;
}
catch (Exception e)
{
SongKong.logger.severe("FinishFail:" + songGroup.getKey());
MainWindow.logger.log(Level.SEVERE, "SongPrematcherMatcher:" + e.getMessage(), e);
if (session.getTransaction() != null)
{
session.getTransaction().rollback();
}
return false;
}
catch (Error e)
{
SongKong.logger.severe("FinishFail:" + songGroup.getKey());
MainWindow.logger.log(Level.SEVERE, "SongPrematcherMatcher:" + e.getMessage(), e);
if (session.getTransaction() != null)
{
session.getTransaction().rollback();
}
return false;
}
catch (Throwable t)
{
SongKong.logger.severe("FinishFail:" + songGroup.getKey());
MainWindow.logger.log(Level.SEVERE, "SongPrematcherMatcher:" + t.getMessage(), t);
if (session.getTransaction() != null)
{
session.getTransaction().rollback();
}
return false;
}
finally
{
if(session.isOpen())
{
session.getTransaction().commit();
HibernateUtil.closeSession(session);
}
}
}
private boolean processSongsWithNewMetadata(SongGroup songGroup, List<Song> songs)
{
MainWindow.logger.info("Prematcher:" + songGroup.getKey() + ":totalcount:" + songs.size());
int count = 0;
//Group based on actual metadata only
MetadataGatherer mg = new MetadataGatherer(songs);
for (String album : mg.getAlbums().keySet())
{
List<Song> songsInGrouping = mg.getAlbums().get(album);
count+=songsInGrouping.size();
MainWindow.logger.warning("Prematcher:" + songGroup.getKey() + ":" + album + ":count:" + songsInGrouping.size());
SongGroup sg = SongGroup.createSongGroupForSongs(songGroup, songsInGrouping);
sg.setRandomFolderNoMetadata(false);
sg.setRandomFolder(false);
processRandomFolder(sg, songsInGrouping);
}
List<Song> songsWithNoInfo = new ArrayList<>(mg.getSongsWithNoRelease());
if(songsWithNoInfo.size()>0)
{
count+=songsWithNoInfo.size();
SongGroup sgWithNoInfo = SongGroup.createSongGroupForSongs(songGroup, songsWithNoInfo);
MainWindow.logger.warning("Prematcher:" + songGroup.getKey() + ":NoMetadata:" + ":count:" + songsWithNoInfo.size());
processRandomFolderNoMetadata(sgWithNoInfo, songsWithNoInfo);
}
if(count<songs.size())
{
MainWindow.logger.warning(songGroup.getKey()+":Not all songs have been processed"+songs.size());
Errors.addErrorWithoutStackTrace(songGroup.getKey()+":Not all songs have been processed:"+songs.size());
}
return true;
}
private boolean processRandomFolder(SongGroup songGroup, List<Song> songs)
{
if(UserPreferences.getInstance().isSearchMusicBrainz())
{
MusicBrainzMetadataMatcher.getService().submit(new MusicBrainzMetadataMatcher(songGroup));
}
else if(UserPreferences.getInstance().isSearchDiscogs())
{
if(songGroup.getSubSongGroups().size() > 1)
{
DiscogsMultiFolderSongGroupMatcher.getService().submit(new DiscogsMultiFolderSongGroupMatcher(songGroup));
}
else if(songGroup.getSongIds().size()==1)
{
DiscogsSongMatcher.getService().submit(new DiscogsSongMatcher(songGroup, songGroup.getSongIds().get(0)));
}
else
{
DiscogsSongGroupMatcher.getService().submit(new DiscogsSongGroupMatcher(songGroup));
}
}
else
{
for (Integer songId : songGroup.getSongIds())
{
SongSaver.getService().submit(new SongSaver(songId));
}
}
return true;
}
/**
* Process a group of files that are in a Random folder and dont seem to have anything in common so should not be grouped
* together.
*
* #param songGroup
* #param songs
* #return
*/
private boolean processRandomFolderNoMetadata(SongGroup songGroup, List<Song> songs)
{
if(UserPreferences.getInstance().isSearchMusicBrainz())
{
for (Song song : songs)
{
MusicBrainzSongMatcher.getService().submit(new MusicBrainzSongMatcher(songGroup, song.getRecNo()));
}
}
else if(UserPreferences.getInstance().isSearchDiscogs())
{
for (Song song : songs)
{
DiscogsSongMatcher.getService().submit(new DiscogsSongMatcher(songGroup, song.getRecNo()));
}
}
else
{
for (Integer songId : songGroup.getSongIds())
{
SongSaver.getService().submit(new SongSaver(songId));
}
}
return true;
}
}

I thought it was something silly but couldn't find it, however the issue was I submitted the tasks to the incorrect ExecutorService
In AcoustidMatcher I had
private boolean processFolderWithPoorMetadata(SongGroup songGroup)
{
MusicBrainzMetadataMatcher.getService().submit(new SongPrematcherMatcher(songGroup));
return true;
}
when I should have had
private boolean processFolderWithPoorMetadata(SongGroup songGroup)
{
SongPrematcherMatcher.getService().submit(new SongPrematcherMatcher(songGroup));
return true;
}
So this means all the SongPrematcher tasks were submitted to the wrong ExecutorService so when I requested to close the SongPrematcher ExecutorService it could be closed down immediately !

Working with BlockingQueue and Multithreads. All Threads Stuck in Waiting

I am creating a system that will have multiple suite deployments and each deployment will have a queue of test suites. Since I want the test suites to run concurrently on their individual suite deployment, I need to add concurrency to the code. I have created a simplified version of the code I am using, but the concurrency portion doesn't work when I try to shut it down.
When the Runner.stopEverything() gets called, the result is that the queue gets emptied, and it waits for the threads to complete, but even when the tests all complete, the wait never finishes even with the notifyAll(). The result is that the process just sits there never ending. I go look at it in debug mode and the result is that all 3 threads show waiting.
Main:
public static void main(String args[]) throws Exception {
Runner.queueTestSuites("SD1", Arrays.asList("A", "B", "C"));
Runner.queueTestSuites("SD2", Arrays.asList("D", "E", "F"));
Runner.queueTestSuites("SD3", Arrays.asList("G", "H", "I"));
Thread.sleep(5000);
System.out.println("~~~~~~~~~~~~~~~~~~~~~~~~");
Runner.stopEverything();
}
Runner:
public class Runner {
private static Map<String, TestQueue> runnerQueueMap = new ConcurrentHashMap<>();
public synchronized static void queueTestSuites(String suiteDeployment, List<String> testSuiteQueueAsJSON) throws Exception {
TestQueue queue;
if(runnerQueueMap.containsKey(suiteDeployment)) {
queue = runnerQueueMap.get(suiteDeployment);
} else {
queue = new TestQueue(suiteDeployment);
}
for (int i = 0; i < testSuiteQueueAsJSON.size(); i++) {
String name = testSuiteQueueAsJSON.get(i);
queue.addToQueue(name);
}
runnerQueueMap.put(suiteDeployment,queue);
}
public synchronized static void stopEverything() throws InterruptedException {
for (String s : runnerQueueMap.keySet()) {
TestQueue q = runnerQueueMap.get(s);
q.saveAndClearQueue();
}
for (String s : runnerQueueMap.keySet()) {
TestQueue q = runnerQueueMap.get(s);
q.waitForThread();
}
System.out.println("All done at " + new Date());
}
}
TestQueue:
public class TestQueue {
private Consumer consumer;
private Thread consumerThread;
private java.util.concurrent.BlockingQueue<String> queue;
private String suiteDeployment;
public TestQueue(String suiteDeployment) {
this.suiteDeployment = suiteDeployment;
queue = new ArrayBlockingQueue<>(100);
startConsumer();
}
public void addToQueue(String testSuite) {
try {
queue.put(testSuite);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
public synchronized void waitForThread() {
try {
if (consumer.running.get()) {
synchronized (consumerThread) {
System.out.println("Waiting for " + consumerThread.getName());
consumerThread.wait();
}
}
System.out.println("Thread complete at " + new Date());
} catch (InterruptedException e) {
e.printStackTrace();
}
}
public void saveAndClearQueue() {
List<String> suiteNames = new ArrayList<>();
for (String suite : queue) {
suiteNames.add(suite);
}
queue.clear();
}
private void startConsumer() {
consumer = new Consumer(queue,suiteDeployment);
consumerThread = new Thread(consumer);
consumerThread.start();
}
private class Consumer implements Runnable{
private BlockingQueue<String> queue;
private String suiteDeployment;
public AtomicBoolean running;
public Consumer(BlockingQueue<String> queue, String suiteDeployment){
this.queue = queue;
this.suiteDeployment = suiteDeployment;
this.running = new AtomicBoolean(false);
}
#Override
public void run() {
try{
while(!Thread.currentThread().isInterrupted()) {
String testSuite = queue.take();
this.running.set(true);
new Test(testSuite, suiteDeployment).run();
this.running.set(false);
}
notifyAll();
}catch(Exception e) {
e.printStackTrace();
}
}
}
}
Test:
public class Test {
String testSuite = "";
String suiteDeployment = "";
public Test(String testSuite, String suiteDeployment) {
this.testSuite = testSuite;
this.suiteDeployment = suiteDeployment;
}
public void run() {
int time = new Random().nextInt() % 10000;
time = Math.max(time, 3000);
System.out.println("Test Started: " + testSuite + " on " + suiteDeployment + " at " + new Date() + " running for " + time + " on thread " + Thread.currentThread().getName());
try {
Thread.sleep(time);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Test Completed: " + testSuite + " on " + suiteDeployment + " at " + new Date());
}
}

Inside run method of your consumer, you have a blocking call to queue.take() which means it will block until there is an item inside your queue. You run out of elements inside the queue eventually and all your thread are blocked by the queue.take() call waiting for more elements to become available to process.
Although your call is in a while loop where it check if the thread is interrupted, you actually never interrupt the threads so it never gets to the while loop evaluation & blocked at the call to queue.take()
So your threads stay in wait as they are waiting for input to become avilable inside your blocking queue
Also your saveAndClear method must lock on the correct object which is the queue itself, like below:
public void saveAndClearQueue() {
List<String> suiteNames = new ArrayList<String>();
synchronized (queue) {
for (String suite : queue) {
suiteNames.add(suite);
}
queue.clear();
}
System.out.println("Saved(not executed) : "+suiteNames);
}
And your waitForThread method should do sth like below:
public void waitForThread() {
synchronized (consumerThread) {
while (consumer.running.get()) {
try {
consumerThread.wait(100);
} catch (InterruptedException e) {
break;
}
}
}
if (!consumer.running.get()) {
consumerThread.interrupt();
}
System.out.println("Thread complete at " + new Date());
}