Develop Reference

Multithreaded Java job deadlock on DB insert

I am experiencing deadlock when running newly implemented job. I'll try to insert all minimum relevant parts of the code
Job
final BlockingQueue<Runnable> BLOCKING_QUEUE = new ArrayBlockingQueue<>(10);
final ThreadPoolExecutor EXECUTOR = new ThreadPoolExecutor(10, 10, 60, TimeUnit.HOURS, BLOCKING_QUEUE, new BlockWhenQueueFull());
protected void doExecute() {
records = service.getBatchRecordsToProcess(batchSize); // Load 500 records from DB by main thread
for (Record record: records) {
EXECUTOR.execute(() -> { // Take thread out of pool and make it process the record
service.processRecord(record);
});
}
}
}
class BlockWhenQueueFull implements RejectedExecutionHandler {
public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
try {
Thread.sleep(200); // If all threads are busy then put main thread to sleep and after 200 ms try once again
} catch (InterruptedException interruptedException) {
LOGGER.debug(interruptedException.getMessage(), interruptedException);
}
executor.execute(r);
}
}
Service
public List<Record> getBatchRecordsToProcess(int batchSize) {
return dao.getRecordsToProcess(batchSize);
}
public void processRecord(Record record) {
processContent(record);
writeToFs(record);
markAsProcessed(record);
}
public void processContent(Record record) {
// business logic
NewRecord newRecord = new NewRecord(record);
dao.insertNewRecord(newRecord);
}
private synchronized void writeToFs(Record record) throws IOException {
// Create folders and files on FS and write there contents
}
private void markAsProcessed(Record record) {
dao.markRecordAsProcessed(record);
}
DAO
#Transactional(readOnly = true)
public List<Record> getRecordsToProcess(int batchSize) {
return jdbcTemplate.query(SELECT_BATCH,
new Object[]{batchSize},
new RecordMapper());
}
#Transactional
public void insertNewRecord(NewRecord newRecord) {
jdbcTemplate.update(
INSERT_NEW_RECORD,
newRecord.getDate(),
// values
);
}
#Transactional
public void markRecordAsProcessed(Record record) {
jdbcTemplate.update(UPDATE_PROCESSED, record.getId());
}
Thread Dump
First state applies for all 10 Threads.
"pool-2-thread-10#8925" prio=5 tid=0x44 nid=NA waiting for monitor entry
java.lang.Thread.State: BLOCKED
waiting for main#1 to release lock on <0x27c9> (a java.util.concurrent.ConcurrentHashMap)
...
at com.dao.NewRecordDao$$EnhancerBySpringCGLIB$$ec63bd65.insertContent(<generated>:-1)
"main#1" prio=5 tid=0x1 nid=NA sleeping
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Thread.java:-1)
at com.job.BlockWhenQueueFull.rejectedExecution(Job.java:130) *pointing to Thread.sleep(200);
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at com.job.BlockWhenQueueFull.rejectedExecution(Job.java:134) *pointing to executor.execute(r);
After the execution the job processes usually couple thousands of records and gets stuck - the main Thread just waits via RejectedExecutionHandler because all worker Threads are stuck. It looks like the worker threads are locked on newRecordDao.insertContent(newRecord); but the table does not have any foreign keys and no table is referencing columns in this table. Index is just on PK.
The question is - Is it possible to get a deadlock on Oracle table when multiple threads are concurrently inserting data (NewRecords), but no reads / updates are performed on this table? The table does not have generated PK - records are inserted with id in a random order (source table has tens of million of records so I do not want to perform costly ordering because of performance). Isolation level appears to be READ COMMITTED.
Is it required for method which writes onto filesystem (tmpfs / xfs) to be synchronized?
Initially I thought it would be problem of synced method for writing to filesystem but if I get the Thread Dump right it is an issue with insert into DB. I have no idea why should the DB inserts be the problem in this case.

Terminate threads in a thread pool for Multiple Tenant

I'm creating a Threadpool as shown below for a job.
public class MoveToCherwellThreadPool {
public static ThreadPoolExecutor cherwellMoveThreadPoolExecutor = null;
private static EMLogger logger = EMLogger.getLogger();
private static final String CLASSNAME = "MoveToCherwellThreadPool";
public static void initiateCherwellMoveThreadPool() {
BlockingQueue<Runnable> q = new LinkedBlockingQueue<Runnable>(100000);
cherwellMoveThreadPoolExecutor = new ThreadPoolExecutor(10,20, 20, TimeUnit.SECONDS, q);
cherwellMoveThreadPoolExecutor.setRejectedExecutionHandler(new RejectedExecutionHandler() {
#Override
public void rejectedExecution(Runnable r,
ThreadPoolExecutor executor) {
logger.logDebug(CLASSNAME,"Rejected task cherwellMoveThreadPoolExecutor Active tasks : " + cherwellMoveThreadPoolExecutor.getActiveCount() + ", " + "cherwellMoveThreadPoolExecutor Completed tasks : " + cherwellMoveThreadPoolExecutor.getCompletedTaskCount()+" Waiting for a second !! ");
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
executor.execute(r);
}
});
}
}
I'm using this during a process running for multiple customers. For each customer new threadpool will be initialized and threads will be running.
Below is the code where I'm using the threadpool.
for (Object[] objects : relationshipList) {
CherwellRelationshipMoveThread relationshipThread = new CherwellRelationshipMoveThread(objects,
this.customerId, sb, credential,mainCIId,moveUniqueId,this.startTime);
CompletableFuture<?> future = CompletableFuture.runAsync(relationshipThread,
MoveToCherwellThreadPool.cherwellMoveThreadPoolExecutor);
crelationshipList.add(future);
}
crelationshipList.forEach(CompletableFuture::join);
This thread will be created for multiple customers. I'm giving an option to terminate this job in UI. On click of stop process I need to stop/kill only the threads running for that particular customer and other customer's thread shouldn't be harmed and should be keep running.
On click of stop process from UI I'm calling a service where inside the service my code will be
MoveToCherwellThreadPool.cherwellMoveThreadPoolExecutor.shutdownNow();
I'm calling shutdownNow() on the ThreadPoolExecutor.
This is killing all the threads of all the customers. I don't want to kill all the customers process, but only for the customer where I'll click on stop process.

This code doesn't maintain any mapping from a tenant to a thread pool, there's only one static reference to a ThreadPoolExecutor. Each time initiateCherwellMoveThreadPool is called, any existing executor will be replaced with a new one, and the existing one isn't shut down, so it will leak resources. As a result, this will execute tasks from multiple tenants in the same thread pool.
This code is also not thread safe. It's possible (if unlikely) that a thread could schedule a task on a newly-created executor, or even shut it down, before setRejectedExecutionHandler is called.
If you need a separate executor per tenant, this will need to be implemented. A good option might be to use a ConcurrentHashMap with customerId keys and ThreadPoolExecutor values, for example (logging omitted for brevity):
public class MoveToCherwellThreadPool {
public static ConcurrentMap<String, ThreadPoolExecutor> cherwellMoveThreadPoolExecutors = new ConcurrentHashMap<>();
public static ThreadPoolExecutor getCherwellMoveThreadPool(String customerId) {
return cherwellMoveThreadPoolExecutors.computeIfAbsent(customerId, id -> {
BlockingQueue<Runnable> q = new LinkedBlockingQueue<Runnable>(100000);
ThreadPoolExecutor executor = new ThreadPoolExecutor(10, 20, 20, TimeUnit.SECONDS, q);
executor.setRejectedExecutionHandler(new RejectedExecutionHandler() { /*...*/ });
return executor;
});
}
public static List<Runnable> stopCherwellMoveTheadPool(String customerId) {
if (cherwellMoveThreadPoolExecutors.containsKey(customerId)) {
return cherwellMoveThreadPoolExecutors.get(customerId).shutdownNow();
}
return Collections.emptyList();
}
}
This can be used like this:
CompletableFuture<?> future = CompletableFuture.runAsync(relationshipThread,
MoveToCherwellThreadPool.getCherwellMoveThreadPool(customerId));
It's also important to realise that calling shutdownNow can only attempt to cancel currently executing tasks, and "does not wait for actively executing tasks to terminate":
This implementation cancels tasks via Thread.interrupt(), so any task that fails to respond to interrupts may never terminate.
The code implementing CherwellRelationshipMoveThread isn't shown, so this may or may not be the case.

Creating a Thread that does not exit

I was wondering what the best way to create a Java Thread that does not terminate.
Currently, I basically have a "Runner" that basically looks like:
ExecutorService pool = Executors.newFixedThreadPool(3);
for (int i = 0; i < numThreads; ++i) {
pool.submit(new Task());
}
pool.shutdown();
and Task looks something like this
public class Task {
...
public void run() {
while(true) { }
}
}
There are two concerns I have with my approach:
Should I be creating a task that just returns after doing work and continue spawning threads that do minimal amounts of work? I'm concerned about the overhead, but am not sure how to measure it.
If I have a Thread that just loops infinitely, when I force quit the executable, will those Threads be shutdown and cleaned up? After some testing, it doesn't appear an InterruptException is being thrown when the code containing the ExecutorService is forcefully shutdown.
EDIT:
To elaborate, the Task looks like
public void run() {
while(true) {
// Let queue be a synchronized, global queue
if (queue has an element) {
// Pop from queue and do a very minimal amount of work on it
// Involves a small amount of network IO (maybe 10-100 ms)
} else {
sleep(2000);
}
}
}

I agree with #D Levant, Blocking queue is the key to use here. With blocking queue, you don't need to handle the queue-empty or queue-full scenario.
In your Task class,
while(true) {
// Let queue be a synchronized, global queue
if (queue has an element) {
// Pop from queue and do a very minimal amount of work on it
// Involves a small amount of network IO (maybe 10-100 ms)
} else {
sleep(2000);
}
}
Its really not a good approach, its inefficient because your while loop is continuously polling, even you have put the thread sleep(), but still its also a overhead of unnecessary context-switches every time the thread wake-ups and sleeps.
In my opinion, your approach of using Executors is looking good for your case. Thread creation is obviously a costly process, and Executors provide us the flexibility of re-using the same thread for different tasks.
You can just pass your task through execute(Runnable) or submit(Runnable/Callable) and then rest will be taken care by Executors internally. Executors internally uses blocking queue concept only.
You can even create your own thread pool by using the ThreadPoolExecutor class and passing the required parameter in its constructor, here you can pass your own blocking queue to hold the tasks. Rest thread-management will be taken care by it on basis of the configuration passes in constructor, So If you are really confident about the configuration parameters then you can go for it.
Now the last point, If you don't want to use the Java's in-built Executors framework, then you can design your solution by using BlockingQueue to hold tasks and starting a thread which will take the tasks from this blocking queue to execute, Below is the high-level implementation:
class TaskRunner {
private int noOfThreads; //The no of threads which you want to run always
private boolean started;
private int taskQueueSize; //No. of tasks that can be in queue at a time, when try to add more tasks, then you have to wait.
private BlockingQueue<Runnable> taskQueue;
private List<Worker> workerThreads;
public TaskRunner(int noOfThreads, int taskQueueSize) {
this.noOfThreads = noOfThreads;
this.taskQueueSize = taskQueueSize;
}
//You can pass any type of task(provided they are implementing Runnable)
public void submitTask(Runnable task) {
if(!started) {
init();
}
try {
taskQueue.put(task);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
public void shutdown() {
for(Worker worker : workerThreads){
worker.stopped = true;
}
}
private void init() {
this.taskQueue = new LinkedBlockingDeque<>(taskQueueSize);
this.workerThreads = new ArrayList<>(noOfThreads);
for(int i=0; i< noOfThreads; i++) {
Worker worker = new Worker();
workerThreads.add(worker);
worker.start();
}
}
private class Worker extends Thread {
private volatile boolean stopped;
public void run() {
if(!stopped) {
try {
taskQueue.take().run();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
}
class Task1 implements Runnable {
#Override
public void run() {
//Your implementation for the task of type 1
}
}
class Task2 implements Runnable {
#Override
public void run() {
//Your implementation for the task of type 2
}
}
class Main {
public static void main(String[] args) {
TaskRunner runner = new TaskRunner(3,5);
runner.submitTask(new Task1());
runner.submitTask(new Task2());
runner.shutdown();
}
}

Waiting for an unknown number of asynchronous tasks

I have a queue of tasks, and a thread that peek the queue once in a few seconds and if there is a task it performs it.
I have another code section (in another thread of course), that creates tasks in a loop (I can't know the number of tasks in advance from outside the loop) and insert them to the queue. The tasks contains some 'result' object, and the external thread (which created those tasks) need to wait for all the tasks to finish and finally get the result from each one of them.
The problem is that I can't pass java Semaphore\CountDownLatch etc to the result object since I don't know the number of monitors in advance.
I also can't use an Executor that uses invokeAll or wait for the Future object since the tasks are unsynchrnized (the external thread just pust the task to a queue and another thread will execute the task when he have time for this).
The only solution I've had in mind is to create some 'Inverted Semaphore' class that holds a set of results and a monitors counter. The getResult function will check if the counter == 0 and if the answer is yes will notify some lock object, and the getResult function will wait for this lock:
public class InvertedSemaphore<T> {
Set<T> resultSet;
int usages;
final Object c;
public InvertedSemaphore() {
resultSet = Collections.synchronizedSet(new HashSet<T>());
usages = 0;
c = new Object();
}
public void addResult(T result) {
resultSet.add(result);
}
public void addResults(Set<T> result) {
resultSet.addAll(result);
}
public void acquire() {
usages++;
}
public void release() {
synchronized (c) {
if (--usages == 0) {
c.notify();
}
}
}
public Set<T> getResults() {
synchronized (c) {
try {
while (usages > 0) {
c.wait();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
return resultSet;
}
}
Each addTask method will invoke semaphore.acquire, and each of the (unsynchronized) tasks will invoke semaphore.release in the end of the task.
It sounds pretty complicated and I'm pretty sure there is a better solution for this in java concurrent library or something.
Any idea will be appriciated:)

If the tasks don't need to be processed in order, use an ExecutorCompletionService
More generally, it is not necessary to use invokeAll on an ExecutorService in order to get a Future for the result. ExecutorService#submit could be used for this purpose, or optionally, the task being created could implement Future itself, thus allowing the creator of the task to ask for the result at a later point in time.
Some code:
class MyTask {
AtomicReference<?> result = new AtomicReference<?>();
void run() {
//do stuff here
result.set(/* the result of the calculation */);
}
boolean resultReady() {
return result.get()!=null;
}
? get() {
return result.get();
}
}
... elsewhere in code
void createTasks() {
Collection<MyTask> c = new ...;
while(indeterminable condition) {
MyTask task = new MyTask();
c.add(task);
mysteryQueue.add(task);
}
while(haven't received all results) {
MyTask task = c.get(...); //or iterate or whatever
? result = task.get();
if (result!=null) {
//do stuff, probably remove the task from the collection c would be smart
}
}
}

One idea would be to use a separate queue for the results.
So you will have one blocking queue that thread A places tasks for thread B thereby having a producer-consumer approach, and when each task is completed, the result could be placed in the second result queue inverting the consumer-producer roles since now thread A that originally created the tasks will consume the result from the second queue.

You can do the following:
each producer will hold its own queue. The producer will pass a means to report to this queue to the Task itself. When the task finishes running, it will queue its result to this queue. It is beast described by some code:
class Result{}
interface IResultCallback{
void resultReady(Result r); // this is an abstraction of the queue
}
class Producer implements IResultCallback{
// the producer needs to pass itself to the constructor of the task,
// the task will only see its "resultReady" facade and will be able to report to it.
// the producer can aggragte the results at it will and execute its own computation as
// as soon it is ready
Queue<Result> results; // = init queue
#Override
public void resultReady(Result r) {
results.add(r);
if(results.size() == 9){
operate();
}
results.clear();
}
public void operate(){
// bla bla
}
}
public class Task {
IResultCallback callback;
public Task(IResultCallback callback){
this.callback = callback;
}
public void execute(){
// bla bla
Result r = null; // init result;
callback.resultReady(r);
}
}

Wait until any of Future<T> is done

I have few asynchronous tasks running and I need to wait until at least one of them is finished (in the future probably I'll need to wait util M out of N tasks are finished).
Currently they are presented as Future, so I need something like
/**
* Blocks current thread until one of specified futures is done and returns it.
*/
public static <T> Future<T> waitForAny(Collection<Future<T>> futures)
throws AllFuturesFailedException
Is there anything like this? Or anything similar, not necessary for Future. Currently I loop through collection of futures, check if one is finished, then sleep for some time and check again. This looks like not the best solution, because if I sleep for long period then unwanted delay is added, if I sleep for short period then it can affect performance.
I could try using
new CountDownLatch(1)
and decrease countdown when task is complete and do
countdown.await()
, but I found it possible only if I control Future creation. It is possible, but requires system redesign, because currently logic of tasks creation (sending Callable to ExecutorService) is separated from decision to wait for which Future. I could also override
<T> RunnableFuture<T> AbstractExecutorService.newTaskFor(Callable<T> callable)
and create custom implementation of RunnableFuture with ability to attach listener to be notified when task is finished, then attach such listener to needed tasks and use CountDownLatch, but that means I have to override newTaskFor for every ExecutorService I use - and potentially there will be implementation which do not extend AbstractExecutorService. I could also try wrapping given ExecutorService for same purpose, but then I have to decorate all methods producing Futures.
All these solutions may work but seem very unnatural. It looks like I'm missing something simple, like
WaitHandle.WaitAny(WaitHandle[] waitHandles)
in c#. Are there any well known solutions for such kind of problem?
UPDATE:
Originally I did not have access to Future creation at all, so there were no elegant solution. After redesigning system I got access to Future creation and was able to add countDownLatch.countdown() to execution process, then I can countDownLatch.await() and everything works fine.
Thanks for other answers, I did not know about ExecutorCompletionService and it indeed can be helpful in similar tasks, but in this particular case it could not be used because some Futures are created without any executor - actual task is sent to another server via network, completes remotely and completion notification is received.

simple, check out ExecutorCompletionService.

ExecutorService.invokeAny

Why not just create a results queue and wait on the queue? Or more simply, use a CompletionService since that's what it is: an ExecutorService + result queue.

This is actually pretty easy with wait() and notifyAll().
First, define a lock object. (You can use any class for this, but I like to be explicit):
package com.javadude.sample;
public class Lock {}
Next, define your worker thread. He must notify that lock object when he's finished with his processing. Note that the notify must be in a synchronized block locking on the lock object.
package com.javadude.sample;
public class Worker extends Thread {
private Lock lock_;
private long timeToSleep_;
private String name_;
public Worker(Lock lock, String name, long timeToSleep) {
lock_ = lock;
timeToSleep_ = timeToSleep;
name_ = name;
}
#Override
public void run() {
// do real work -- using a sleep here to simulate work
try {
sleep(timeToSleep_);
} catch (InterruptedException e) {
interrupt();
}
System.out.println(name_ + " is done... notifying");
// notify whoever is waiting, in this case, the client
synchronized (lock_) {
lock_.notify();
}
}
}
Finally, you can write your client:
package com.javadude.sample;
public class Client {
public static void main(String[] args) {
Lock lock = new Lock();
Worker worker1 = new Worker(lock, "worker1", 15000);
Worker worker2 = new Worker(lock, "worker2", 10000);
Worker worker3 = new Worker(lock, "worker3", 5000);
Worker worker4 = new Worker(lock, "worker4", 20000);
boolean started = false;
int numNotifies = 0;
while (true) {
synchronized (lock) {
try {
if (!started) {
// need to do the start here so we grab the lock, just
// in case one of the threads is fast -- if we had done the
// starts outside the synchronized block, a fast thread could
// get to its notification *before* the client is waiting for it
worker1.start();
worker2.start();
worker3.start();
worker4.start();
started = true;
}
lock.wait();
} catch (InterruptedException e) {
break;
}
numNotifies++;
if (numNotifies == 4) {
break;
}
System.out.println("Notified!");
}
}
System.out.println("Everyone has notified me... I'm done");
}
}

As far as I know, Java has no analogous structure to the WaitHandle.WaitAny method.
It seems to me that this could be achieved through a "WaitableFuture" decorator:
public WaitableFuture<T>
extends Future<T>
{
private CountDownLatch countDownLatch;
WaitableFuture(CountDownLatch countDownLatch)
{
super();
this.countDownLatch = countDownLatch;
}
void doTask()
{
super.doTask();
this.countDownLatch.countDown();
}
}
Though this would only work if it can be inserted before the execution code, since otherwise the execution code would not have the new doTask() method. But I really see no way of doing this without polling if you cannot somehow gain control of the Future object before execution.
Or if the future always runs in its own thread, and you can somehow get that thread. Then you could spawn a new thread to join each other thread, then handle the waiting mechanism after the join returns... This would be really ugly and would induce a lot of overhead though. And if some Future objects don't finish, you could have a lot of blocked threads depending on dead threads. If you're not careful, this could leak memory and system resources.
/**
* Extremely ugly way of implementing WaitHandle.WaitAny for Thread.Join().
*/
public static joinAny(Collection<Thread> threads, int numberToWaitFor)
{
CountDownLatch countDownLatch = new CountDownLatch(numberToWaitFor);
foreach(Thread thread in threads)
{
(new Thread(new JoinThreadHelper(thread, countDownLatch))).start();
}
countDownLatch.await();
}
class JoinThreadHelper
implements Runnable
{
Thread thread;
CountDownLatch countDownLatch;
JoinThreadHelper(Thread thread, CountDownLatch countDownLatch)
{
this.thread = thread;
this.countDownLatch = countDownLatch;
}
void run()
{
this.thread.join();
this.countDownLatch.countDown();
}
}

If you can use CompletableFutures instead then there is CompletableFuture.anyOf that does what you want, just call join on the result:
CompletableFuture.anyOf(futures).join()
You can use CompletableFutures with executors by calling the CompletableFuture.supplyAsync or runAsync methods.

Since you don't care which one finishes, why not just have a single WaitHandle for all threads and wait on that? Whichever one finishes first can set the handle.

See this option:
public class WaitForAnyRedux {
private static final int POOL_SIZE = 10;
public static <T> T waitForAny(Collection<T> collection) throws InterruptedException, ExecutionException {
List<Callable<T>> callables = new ArrayList<Callable<T>>();
for (final T t : collection) {
Callable<T> callable = Executors.callable(new Thread() {
#Override
public void run() {
synchronized (t) {
try {
t.wait();
} catch (InterruptedException e) {
}
}
}
}, t);
callables.add(callable);
}
BlockingQueue<Runnable> queue = new ArrayBlockingQueue<Runnable>(POOL_SIZE);
ExecutorService executorService = new ThreadPoolExecutor(POOL_SIZE, POOL_SIZE, 0, TimeUnit.SECONDS, queue);
return executorService.invokeAny(callables);
}
static public void main(String[] args) throws InterruptedException, ExecutionException {
final List<Integer> integers = new ArrayList<Integer>();
for (int i = 0; i < POOL_SIZE; i++) {
integers.add(i);
}
(new Thread() {
public void run() {
Integer notified = null;
try {
notified = waitForAny(integers);
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
System.out.println("notified=" + notified);
}
}).start();
synchronized (integers) {
integers.wait(3000);
}
Integer randomInt = integers.get((new Random()).nextInt(POOL_SIZE));
System.out.println("Waking up " + randomInt);
synchronized (randomInt) {
randomInt.notify();
}
}
}