Develop Reference

Java Semaphore - permit granted without prior release

I'm writing this question because we faced an scenario in one of the qa environments where it seems like the semaphore failed.
We have only one semaphore:
private Semaphore lock = new Semaphore(1);
It happened that one thread (Quartz job) was running, holding the lock acquired, and then, another job was triggered and got in the middle of the execution.
Both jobs acquire and then release the lock, so if the first one gets delayed, the latter has to wait for the lock to be released by the first one.
The weird part is that the latter didn't wait, it just passed through the lock.acquire()
Scenario it's not complex at all, and the code has been working since the very beginning. We weren't able to recreate it so far, I'm clueless. A glitch maybe?
I'm wondering if someone knows if there is a kind of known incompatibility between Quartz and Java Semaphores, or if Java semaphores could fail under certain scenarios
EDIT
One more detail, it's an app built on Deltaspike CDI framework
This is the Singleton for handling the lock:
import java.util.concurrent.Semaphore;
public class Lock {
private Lock() {}
private static class SingletonHolder {
public static final Lock INSTANCE = new Lock();
}
/**
* Use this method to get a reference to the singleton instance of
* {#link Lock}
*
* #return the singleton instance
*/
public static Lock getInstance() {
return SingletonHolder.INSTANCE;
}
/**
* we allow only one thread at at time
*/
private Semaphore lock = new Semaphore(1);
public void getLock() throws InterruptedException {
lock.acquire();
}
public void releaseLock() {
lock.release();
}
}
This is the first job:
#Scheduled(cronExpression = "{cronExp1}")
public class Job1 implements Job {
#Override
public void execute(JobExecutionContext jobExecutionContext) throws JobExecutionException {
method1();
}
public void method1(){
long threadId = Thread.currentThread().getId();
try{
logger.debug("Thread # " + threadId + "Requesting lock...");
Lock.getInstance().getLock();
logger.debug("Thread # " + threadId + "Lock acquired.");
//...some logic
}catch (PersistenceException e) {
//.. handling exception
}catch (Exception e) {
//.. handling exception
}finally {
Lock.getInstance().releaseLock();
logger.debug("Thread # " + threadId + "Lock released.");
}
}
}
This is the second job:
#Scheduled(cronExpression = "{cronExp2}")
public class Job2 implements Job {
#Override
public void execute(JobExecutionContext jobExecutionContext) throws JobExecutionException {
long threadId = Thread.currentThread().getId();
try{
logger.debug("Thread # " + threadId + "Requesting lock...");
Lock.getInstance().getLock();
logger.debug("Thread # " + threadId + "Lock acquired.");
//...some logic
}catch (PersistenceException e) {
//.. handling exception
}catch (Exception e) {
//.. handling exception
}finally {
Lock.getInstance().releaseLock();
logger.debug("Thread # " + threadId + "Lock released.");
}
}
}
As you can see, the only difference between the jobs (other than the logic), is that Job1 enters the critical zone inside method1, while Job2 does in inside the execute method

Editable queue of tasks running in background thread

I know this question was answered many times, but I'm struggling to understand how it works.
So in my application the user must be able to select items which will be added to a queue (displayed in a ListView using an ObservableList<Task>) and each item needs to be processed sequentially by an ExecutorService.
Also that queue should be editable (change the order and remove items from the list).
private void handleItemClicked(MouseEvent event) {
if (event.getClickCount() == 2) {
File item = listView.getSelectionModel().getSelectedItem();
Task<Void> task = createTask(item);
facade.getTaskQueueList().add(task); // this list is bound to a ListView, where it can be edited
Future result = executor.submit(task);
// where executor is an ExecutorService of which type?
try {
result.get();
} catch (Exception e) {
// ...
}
}
}
Tried it with executor = Executors.newFixedThreadPool(1) but I don't have control over the queue.
I read about ThreadPoolExecutor and queues, but I'm struggling to understand it as I'm quite new to Concurrency.
I need to run that method handleItemClicked in a background thread, so that the UI does not freeze, how can I do that the best way?
Summed up: How can I implement a queue of tasks, which is editable and sequentially processed by a background thread?
Please help me figure it out
EDIT
Using the SerialTaskQueue class from vanOekel helped me, now I want to bind the List of tasks to my ListView.
ListProperty<Runnable> listProperty = new SimpleListProperty<>();
listProperty.set(taskQueue.getTaskList()); // getTaskList() returns the LinkedList from SerialTaskQueue
queueListView.itemsProperty().bind(listProperty);
Obviously this doesn't work as it's expecting an ObservableList. There is an elegant way to do it?

The simplest solution I can think of is to maintain the task-list outside of the executor and use a callback to feed the executor the next task if it is available. Unfortunately, it involves synchronization on the task-list and an AtomicBoolean to indicate a task executing.
The callback is simply a Runnable that wraps the original task to run and then "calls back" to see if there is another task to execute, and if so, executes it using the (background) executor.
The synchronization is needed to keep the task-list in order and at a known state. The task-list can be modified by two threads at the same time: via the callback running in the executor's (background) thread and via handleItemClicked method executed via the UI foreground thread. This in turn means that it is never exactly known when the task-list is empty for example. To keep the task-list in order and at a known fixed state, synchronization of the task-list is needed.
This still leaves an ambiguous moment to decide when a task is ready for execution. This is where the AtomicBoolean comes in: a value set is always immediatly availabe and read by any other thread and the compareAndSet method will always ensure only one thread gets an "OK".
Combining the synchronization and the use of the AtomicBoolean allows the creation of one method with a "critical section" that can be called by both foreground- and background-threads at the same time to trigger the execution of a new task if possible. The code below is designed and setup in such a way that one such method (runNextTask) can exist. It is good practice to make the "critical section" in concurrent code as simple and explicit as possible (which, in turn, generally leads to an efficient "critical section").
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicBoolean;
public class SerialTaskQueue {
public static void main(String[] args) {
ExecutorService executor = Executors.newSingleThreadExecutor();
// all operations on this list must be synchronized on the list itself.
SerialTaskQueue tq = new SerialTaskQueue(executor);
try {
// test running the tasks one by one
tq.add(new SleepSome(10L));
Thread.sleep(5L);
tq.add(new SleepSome(20L));
tq.add(new SleepSome(30L));
Thread.sleep(100L);
System.out.println("Queue size: " + tq.size()); // should be empty
tq.add(new SleepSome(10L));
Thread.sleep(100L);
} catch (Exception e) {
e.printStackTrace();
} finally {
executor.shutdownNow();
}
}
// all lookups and modifications to the list must be synchronized on the list.
private final List<Runnable> tasks = new LinkedList<Runnable>();
// atomic boolean used to ensure only 1 task is executed at any given time
private final AtomicBoolean executeNextTask = new AtomicBoolean(true);
private final Executor executor;
public SerialTaskQueue(Executor executor) {
this.executor = executor;
}
public void add(Runnable task) {
synchronized(tasks) { tasks.add(task); }
runNextTask();
}
private void runNextTask() {
// critical section that ensures one task is executed.
synchronized(tasks) {
if (!tasks.isEmpty()
&& executeNextTask.compareAndSet(true, false)) {
executor.execute(wrapTask(tasks.remove(0)));
}
}
}
private CallbackTask wrapTask(Runnable task) {
return new CallbackTask(task, new Runnable() {
#Override public void run() {
if (!executeNextTask.compareAndSet(false, true)) {
System.out.println("ERROR: programming error, the callback should always run in execute state.");
}
runNextTask();
}
});
}
public int size() {
synchronized(tasks) { return tasks.size(); }
}
public Runnable get(int index) {
synchronized(tasks) { return tasks.get(index); }
}
public Runnable remove(int index) {
synchronized(tasks) { return tasks.remove(index); }
}
// general callback-task, see https://stackoverflow.com/a/826283/3080094
static class CallbackTask implements Runnable {
private final Runnable task, callback;
public CallbackTask(Runnable task, Runnable callback) {
this.task = task;
this.callback = callback;
}
#Override public void run() {
try {
task.run();
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
callback.run();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
// task that just sleeps for a while
static class SleepSome implements Runnable {
static long startTime = System.currentTimeMillis();
private final long sleepTimeMs;
public SleepSome(long sleepTimeMs) {
this.sleepTimeMs = sleepTimeMs;
}
#Override public void run() {
try {
System.out.println(tdelta() + "Sleeping for " + sleepTimeMs + " ms.");
Thread.sleep(sleepTimeMs);
System.out.println(tdelta() + "Slept for " + sleepTimeMs + " ms.");
} catch (Exception e) {
e.printStackTrace();
}
}
private String tdelta() { return String.format("% 4d ", (System.currentTimeMillis() - startTime)); }
}
}
Update: if groups of tasks need to be executed serial, have a look at the adapted implementation here.

Condition instance signalAll() doesn't return

I have the following code:
#Log4j
public class ItemStore {
final Lock lock = new ReentrantLock();
final Condition hasItem = lock.newCondition();
private Map<String, String> store = new Hashtable<>();
public void put( String handle, String item) {
store.put( handle, item );
log.info("stored " + handle );
hasItem.signalAll();
log.info("signaled all threads");
}
public String fetchWithTimeout( String handle, long timeoutInSec ) throws InterruptedException {
try {
lock.lock();
while ( !store.containsKey( handle ) ) {
log.info("store doesn't have " + handle + "; keep waiting");
hasItem.await( timeoutInSec, TimeUnit.SECONDS);
}
return store.get( handle );
} finally {
lock.unlock();
}
}
}
#Test
public void test_withPut() throws InterruptedException {
ItemStore itemStore = new ItemStore();
final String key = "foo";
final String value = "bar";
new Thread() {
#Override
public void run() {
try {
Thread.sleep(3000);
log.info("slept 3 seconds");
itemStore.put(key, value);
} catch (Exception e) {
}
}
}.start();
log.info("fetching");
String actual = itemStore.fetchWithTimeout(key, 20);
log.info("actual = " + actual );
assertEquals( actual, value );
}
Based on the logs from the test as below:
2014-10-05 17:52:48 INFO com.tns.ct.downloader.tests.commons.ItemStoreTest.test_withPut():36 - fetching
2014-10-05 17:52:48 INFO com.tns.ct.downloader.tests.commons.ItemStore.fetchWithTimeout():30 - store doesn't have foo; keep waiting
2014-10-05 17:52:51 INFO com.tns.ct.downloader.tests.commons.ItemStoreTest.run():29 - slept 3 seconds
2014-10-05 17:52:51 INFO com.tns.ct.downloader.tests.commons.ItemStore.put():21 - stored foo
2014-10-05 17:53:08 INFO com.tns.ct.downloader.tests.commons.ItemStoreTest.test_withPut():38 - actual = bar
it seems that hasItem.signalAll() has never returned, as the signaled all threads log has never been issued. Another clue is that the program exited only when the 20 seconds timeout was reached. So, why is the signalAll() method blocked in this case?

Quote from the documentation of signalAll():
An implementation may (and typically does) require that the current thread hold the lock associated with this Condition when this method is called.
Quote from the documentation of ReentrantLock.newCondition():
The returned Condition instance supports the same usages as do the Object monitor methods (wait, notify, and notifyAll) when used with the built-in monitor lock.
If this lock is not held when any of the Condition waiting or signalling methods are called, then an IllegalMonitorStateException is thrown.
Not sure why an IllegalMonitorException isn't thrown in your test, but what's sure is the the putting thread doesn't hold the lock when it calls signalAll() on the condition.
EDIT: as #Fildor mentions, an exception is probably thrown, but swallowed by the empty catch block in your test. Don't use empty catch blocks. If you threw a runtime exception wrapping the caught exception instead of swallowing it, the problem would become obvious.

execute two methods at the same time

I am working on my first multithreaded app in many years. The problem im having is i need to execute two methods at the same time. here's my engine class:
public class ThreadPoolEngine {
// create BlockingQueue to put fund transfer objects
private BlockingQueue<GlobalSearchRequest> searchQueue;
public ThreadPoolExecutor executor;
private HashMap<String, GlobalSearchProcessorCallable> callableMap;
private ArrayList<Future<Integer>> futurList;
Logger logger = Logger.getLogger(ThreadPoolEngine.class);
private Integer gthreadCount;
private Integer gjobPerThread;
public ThreadPoolEngine(Integer threadCount, Integer jobPerThread) {
gthreadCount = threadCount;
gjobPerThread = jobPerThread;
// create a thread pool with the entered no of threads
executor = new HammerThreadPoolExecutor(threadCount, threadCount, 0L,
TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>());
searchQueue = new ArrayBlockingQueue<GlobalSearchRequest>(jobPerThread);
callableMap = new HashMap<String, GlobalSearchProcessorCallable>();
// create list to store reference to Future objects
futurList = new ArrayList<Future<Integer>>();
}
public void createAndSubmitTasks() {
// create Callables
for (int i = 0; i < gthreadCount; i++) {
GlobalSearchProcessorCallable callable1 = new GlobalSearchProcessorCallable(
"SearchProcessor_" + i, searchQueue);
callableMap.put(callable1.getThreadName(), callable1);
// submit callable tasks
Future<Integer> future;
future = executor.submit(callable1);
futurList.add(future);
}
}
public void populateSearchQueue() throws InterruptedException {
// put orderVO objects in BlockingQueue
KeywordFactory key = KeywordFactory.getInstance();
for (int i = 0; i < gjobPerThread*gthreadCount; i++) {
// this method will put SearchRequest object in the order queue
try {
searchQueue.put(new GlobalSearchRequest(key.getRandomPhrase(3)));
} catch (KeywordNoDataFileException e) {
e.printStackTrace();
}
}
}
public void printProcessorStatus() throws InterruptedException {
// print processor status until all orders are processed
while (!searchQueue.isEmpty()) {
for (Map.Entry<String, GlobalSearchProcessorCallable> e : callableMap
.entrySet()) {
logger.debug(e.getKey() + " processed order count: "
+ e.getValue().getProcessedCount());
}
Thread.sleep(1000);
}
}
public void shutDown(boolean forceShutdown) {
if (!forceShutdown) {
// shutdown() method will mark the thread pool shutdown to true
executor.shutdown();
logger.debug("Executor shutdown status " + executor.isShutdown());
logger.debug("Executor terninated status "
+ executor.isTerminated());
// Mark threads to return threads gracefully.
for (Map.Entry<String, GlobalSearchProcessorCallable> orderProcessor : callableMap
.entrySet()) {
orderProcessor.getValue().setRunning(false);
}
} else {
for (Future<Integer> f : futurList) {
f.cancel(true);
}
// shutdown() method will mark the thread pool shutdown to true
executor.shutdownNow();
}
}
public void printWorkersResult() {
for (Future<Integer> f : futurList) {
try {
Integer result = f.get(1000, TimeUnit.MILLISECONDS);
logger.debug(f + " result. Processed orders " + result);
} catch (InterruptedException e) {
logger.error(e.getMessage(), e);
} catch (ExecutionException e) {
logger.error(e.getCause().getMessage(), e);
} catch (TimeoutException e) {
logger.error(e.getMessage(), e);
} catch (CancellationException e) {
logger.error(e.getMessage(), e);
}
}
}
}
ok, i have a main class that instantiates this one and calls two methods on this class, the populateSearchQueue and the createAndSubmitTasks to run my worker class and process the items in the search queue.
The problem
the populateSearchQueue method could take a VERY long time to build (im going to hammer the system with one billion queries at a time) and it could take a lot of memory. Is there a way in java where my main class can call the populateSearchQueue and the createAndSubmitTasks at the same time so the worker threads can start working on the queue while its still being built by the populateSearchQueue method?

I actually solved it. I read my code again and realized that it takes a tiny amount of time to create the thread pool. So call createAndSubmitTasks that creates the thread pool and each assigned a worker class waiting to do something. When that method is done, I now have my pool of 1000 threads sitting there doing nothing. Then the moment I call populateSearchQueue, those worker threads that were sitting idle for the few milliseconds it took to move to the next method, now start grabbing jobs out of the queue, and I get my desired result. The method that is putting stuff in the queue is processing at the same time the worker threads are grabbing jobs out of that queue and running them.
So I reverse the order I'm calling the methods. It's a thing of beauty.

waiting Thread never wakes up

I have a ThreadManager with two Threads. One for gui-relevant requests and one for measurement-relevant requests. The are both running and checking their queue of requests, if there is any, they are processing the request. One can add requests at any time, using the static ThreadManager.addGuiRequest(eGuiRequest) and ThreadManager.addMeasRequest(eMeasRequest) methods. Now both of those need to be initialized which is done by adding a INIT request to the corresponding queue. But the initialization of the measurement is depending on the fact that the gui is already initialized. I tried to solve this using wait()/notify(), but I can not get it working.
Here is a SSCCE. At startup, both queues have a INIT request added and are then started. The measurement initialization detects that the gui is not yet initialized and perfomrs a wait(). The gui initializes (simulated by sleeping for 5s). This all works fine.
After the gui initialized, it tries to wake up the measurement thread, but the measurement thread does not wake up... I based my wait()/notify() code on this article. What is going wrong here?
import java.util.LinkedList;
import java.util.NoSuchElementException;
public class ThreadManager {
public static void main(String[] args) {
new ThreadManager();
ThreadManager.addMeasRequest(eMeasRequest.OTHER_STUFF);
}
public enum eGuiRequest { INIT, OTHER_STUFF; }
public enum eMeasRequest { INIT, OTHER_STUFF; }
private static LinkedList<eGuiRequest> guiQueue = new LinkedList<eGuiRequest>();
private static LinkedList<eMeasRequest> measQueue = new LinkedList<eMeasRequest>();
private static Thread guiThread, measThread;
protected boolean initialized = false;
public ThreadManager() {
final int waitMs = 200;
guiThread = new Thread(new Runnable() {
#Override
public void run() {
while (true) {
try {
if (guiQueue.isEmpty()) sleepMs(waitMs);
else {
eGuiRequest req = guiQueue.getFirst();
processGuiRequest(req);
guiQueue.removeFirst();
}
} catch (NoSuchElementException e) {}
}
}
private void processGuiRequest(eGuiRequest req) {
System.out.println("T: " + "Processing Gui request: " + req);
switch (req) {
case INIT:
// do some initializiation here - replaced by a wait:
sleepMs(5000);
System.out.println("I: " + "guiThread finished, waking up measThread");
synchronized (measThread) {
initialized = true;
measThread.notify();
}
break;
case OTHER_STUFF:
// do other stuff
break;
}
}
});
measThread = new Thread(new Runnable() {
#Override
public void run() {
while (true) {
try {
if (measQueue.isEmpty()) sleepMs(waitMs);
else {
eMeasRequest req = measQueue.getFirst();
processMeasurementRequest(req);
measQueue.removeFirst();
}
} catch (NoSuchElementException e) {}
}
}
private void processMeasurementRequest(eMeasRequest req) {
if (req == eMeasRequest.INIT) { // if init, wait until GUI is initialized
synchronized (this) {
while (!initialized) {
System.out.println("I: " + "measThread waits for guiThread to finish initializiation");
try {
wait();
} catch (Exception e) {}
System.out.println("I: " + "measThread awakes");
}
}
}
System.out.println("T: " + "Processing Measurement request: " + req);
// process request here:
sleepMs(5000);
}
});
addGuiRequest(eGuiRequest.INIT);
addMeasRequest(eMeasRequest.INIT);
guiThread.start();
measThread.start();
}
public static void sleepMs(int ms) {
try {
Thread.sleep(ms);
} catch (InterruptedException ee) {}
}
public static void addGuiRequest(eGuiRequest req) {
guiQueue.add(req);
}
public static void addMeasRequest(eMeasRequest req) {
measQueue.add(req);
}
}

The GUI thread calls notify() on measThread (of type Thread), and the processMeasurementRequest() method calls wait() on this, which is the Runnable instance used by measThread.
I would advise using a specific object, shared by both threads to wait and notify:
private static final Object GUI_INITIALIZATION_MONITOR = new Object();
Also, instead of using a LinkedList and sleeping an aritrary time between requests, I would use a BlockingQueue: this would allow the consuming thread to handle a request as soon as there is one, and would avoid unnecessary wakeups from the sleeping state.
Also, instead of the low-level wait/notify, you could use a CountDownLatch initialized to 1. The GUI thread would countDown() the latch when it's initialized, and the mesurement thread would await() the latch until the GUI thread has called countDown(). This would delegate complex synchronization and notification stuff to a more high-level, well-tested object.

The main problem is that you call notify() on measThread, but wait() is called on an anonymous class. The easiest way to fix this is to create a special object for synchronization. For example, you create a field:
private static final Object LOCK = new Object();
Then you write synchronized blocks using this object and call its methods like this:
synchronized (LOCK) {
while (!initialized) LOCK.wait();
}
Also I have to say that this piece of code doesn't use any synchronization at all for the fields accessed from different threads, which means that it can break at any time. Both queues are accessed outside the threads created by you, this means that you should either access them with a lock held all the time, or you can make them thread safe by using a built-in synchronized list:
quiQueue = Collections.synchronizedList(new LinkedList<eGuiRequest>());
initialized is accessed from synchronized blocks, but right now they synchronize on different locks (I have described this problem at the start of my answer). If you fix this problem, initialized will also be working as it should.

Just do not sent init request to measurment at startup. Sent it from processGuiRequest() after execution of init gui request. Then no wait/notify stuff is needed.