Develop Reference

Java thread hung in a native method

While parsing a date string the thread hung in the native method. The thread is stopped in the same state for the last 18 hours(Thread dumb in different times shows the same show the same stack for the thread in question).
Please suggest what might be the error
"DateParsingThread - 001" #228 daemon prio=5 os_prio=0 tid=0x00007f98e0029000 nid=0x9c2 runnable [0x00007f98016ee000]
java.lang.Thread.State: RUNNABLE
at java.security.AccessController.doPrivileged(Native Method)
at sun.util.resources.LocaleData.getBundle(LocaleData.java:163)
at sun.util.resources.LocaleData.getDateFormatData(LocaleData.java:127)
at sun.util.locale.provider.LocaleResources.getCalendarNames(LocaleResources.java:325)
at sun.util.locale.provider.CalendarNameProviderImpl.getDisplayNamesImpl(CalendarNameProviderImpl.java:157)
at sun.util.locale.provider.CalendarNameProviderImpl.getDisplayNames(CalendarNameProviderImpl.java:139)
at sun.util.locale.provider.CalendarDataUtility$CalendarFieldValueNamesMapGetter.getObject(CalendarDataUtility.java:178)
at sun.util.locale.provider.CalendarDataUtility$CalendarFieldValueNamesMapGetter.getObject(CalendarDataUtility.java:154)
at sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObjectImpl(LocaleServiceProviderPool.java:281)
at sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObject(LocaleServiceProviderPool.java:265)
at sun.util.locale.provider.CalendarDataUtility.retrieveFieldValueNames(CalendarDataUtility.java:88)
at java.util.Calendar.getDisplayNames(Calendar.java:2178)
at java.text.SimpleDateFormat.getDisplayNamesMap(SimpleDateFormat.java:2366)
at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:1972)
at java.text.SimpleDateFormat.parse(SimpleDateFormat.java
at java.text.DateFormat.parse(DateFormat.java:364)
Any suggestions to find the solutions are welcome.

Java / Wildfly: Threads piling up / Eclipse shows one additional running thread for each request

I am using javas Thread to connect via SMTP to our mailprovider as this can take some time until it finishes and I dont want the request to wait.
But it looks like the threads are not closed after they are finished.
I noticed this in the debug mode of Eclipse:
For each time I create a new Thread(), it adds one running thread, but it is not closing it (at least I assume this, as eclipse still shows Running).
This is my code:
Thread mailThread = new Thread() {
public void run() {
System.out.println("Does it work?");
try {
Transport t = session.getTransport("smtp");
t.connect("user","pass");
t.sendMessage(message,message.getAllRecipients());
t.close();
System.out.println("SENT");
return;
} catch (MessagingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return;
}
}
};
mailThread.start();
Is this working as intended? Or does Running in eclipse mean something different?

I suggest not only to use the debugger to see, to see which threads you have at a certain point in time. Debuggers might display threads which are active during a break point but should not be there under normal conditions.
It is preferrable to use the command line tool jstack to create thread dumps. This will dump all the threads in a JVM at a certain point in time.
Here are some instructions on how to use it: https://helpx.adobe.com/uk/experience-manager/kb/TakeThreadDump.html
Another thing could help you debugging and finding threads in the dump: give threads a name using the string in one of the constructor.
new Thread("foo")
Then it becomes easier to find these in the thread dump.
If you call a thread "foo" then it will show up in a thread dump like this:
"foo" #16 prio=5 os_prio=0 tid=0x0000000041970800 nid=0x41f8 waiting on condition [0x000000004244e000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base#9/Native Method)
at stackoverflow.ThreadReferenceTest$1.run(ThreadReferenceTest.java:14)
Locked ownable synchronizers:
- None
"Service Thread" #15 daemon prio=9 os_prio=0 tid=0x0000000041914000 nid=0x3d90 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None

Why doesn't my program stop even when i call disruptor.shutdown

i have been try to use the LMAX distruptor to buffer the content produced by one of my programs and publish them to another program as a batch of records (well i am still unable to get the consumer batching part done). But even without using the batching of the records, it works as it should be. But my problem is eventhough i used call the
`disruptor.shutdown()` and `executorService.shutdownNow()`
as it is given in one of the examples, it doesn't stop executing the program. It does even execute statement below those methods as well. When i print
executorService.isShutdown();
it returns true. Can someone help me with this...
Edit
"pool-1-thread-1" prio=10 tid=0x00007f57581b9800 nid=0x1bec waiting on condition [0x00007f573eb0d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d9110148> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45)
at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Some tips that help me:
1. Set daemon flag on ExecutorService threads
Do this with a ThreadFactory (or Guava's ThreadFactoryBuilder).
Example:
final ThreadFactory threadFactory =
new ThreadFactory() {
#Override
public Thread newThread(Runnable r) {
final ThreadFactory threadFactory = Executors.defaultThreadFactory();
final Thread thread = threadFactory.newThread(r);
thread.setDaemon(true);
return thread;
}
};
final ExecutorService executorService =
Executors.newFixedThreadPool(threadCount, threadFactory);
2. Shutdown order
Disruptor.shutdown(long, TimeUnit)
Disruptor.halt()
ExecutorService.shutdown()
ExecutorService.awaitTermination(long, TimeUnit)
Impatient shutdown example:
try {
disruptor.shutdown(0, TimeUnit.NANOSECONDS);
// if shutdown is successful:
// 1. exception is not thrown (obviously)
// 2. Disruptor.halt() is called automatically (less obvious)
}
catch (TimeoutException e) {
disruptor.halt();
}
executorService.shutdown();
executorService.awaitTermination(0, TimeUnit.NANOSECONDS);
3. Use a Shutdown Hook
These are called even when System.exit(int) is called, but not if your JVM is killed with SIGKILL (or the equivalent on non-POSIX platforms).
Runtime.getRuntime()
.addShutdownHook(
new Thread(
() -> {
// shutdown here
}));

Your Java Process only stops, when all threads (that are non daemon threads) are finished.
Probably some thread is still running, maybe in a lock, maybe in a loop.
To see what thread are still running you can use the jdk-tools:
Use jps to get the ids of running Java processes:
C:\DVE\jdk\jdk8u45x64\jdk1.8.0_45\bin>jps
4112 TestExMain
With the right id for your program use the command jstack:
C:\DVE\jdk\jdk8u45x64\jdk1.8.0_45\bin>jstack 4112
2015-09-17 09:12:45
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode):
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x000000001d208800 nid=0x1b7c runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"main" #1 prio=5 os_prio=0 tid=0x0000000002260800 nid=0x1324 waiting on condition [0x000000000224f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.example.TestExMain.main(TestExMain.java:8)
For example, here you'll see a thread Service Thread which is a daemon - this thread wont stop your program from shutting down.
The Thread main is not a daemon thread - the Java-Process will wait for this thread to finish, before it stops.
For each thread you'll see a stack trace, at which position the thread is - with that you can find the code that possible keeps the thread from running.
That specific Thread you have there is somehow locked (im not shure why, it could be a wait() call, a synchronize block or some other locking mechanism). When that thread doesn't stop when you call disruptor.shutdown() it might be a bug in that lib you use.

Java Deadlock during a synchronized on a local resource?

I'm seeing a problem with multiple Threads deadlocking on the same line of code.
I cannot reproduce the problem locally or in any test, but yet Thread Dumps from Production have show the problem quite clearly.
I can't see why the Threads would become blocked on the synchronized line below, since there is no other synchronization on the Object in the call stack or in any other Thread. Does anyone have any idea what is going on, or how I can even reproduce this issue (Currently trying with 15 Threads all hitting trim() in a loops, while processing 2000 tasks through my Queue - But unable to reproduce)
In the Thread dump below, I think the multiple Threads with the 'locked' status may be a manifestation of Java Bug: http://bugs.java.com/view_bug.do?bug_id=8047816 where JStack reports Threads in wrong state.
(I'm using JDK Version: 1.7.0_51)
Cheers!
Here is a view of the Threads in the Thread dump.....
"xxx>Job Read-3" daemon prio=10 tid=0x00002aca001a6800 nid=0x6a3b waiting for monitor entry [0x0000000052ec4000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.mycompany.collections.CustomQueue.remove(CustomQueue.java:101)
- locked <0x00002aae6465a650> (a java.util.ArrayDeque)
at com.mycompany.collections.CustomQueue.trim(CustomQueue.java:318)
at com.mycompany.collections.CustomQueue.itemProcessed(CustomQueue.java:302)
at com.mycompany.collections.CustomQueue.trackCompleted(CustomQueue.java:147)
at java.util.concurrent.ThreadPoolExecutor.afterExecute(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- <0x00002aaf5f9c2680> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"xxx>Job Read-2" daemon prio=10 tid=0x00002aca001a5000 nid=0x6a3a waiting for monitor entry [0x0000000052d83000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.mycompany.collections.CustomQueue.remove(CustomQueue.java:101)
- locked <0x00002aae6465a650> (a java.util.ArrayDeque)
at com.mycompany.collections.CustomQueue.trim(CustomQueue.java:318)
at com.mycompany.collections.CustomQueue.itemProcessed(CustomQueue.java:302)
at com.mycompany.collections.CustomQueue.trackCompleted(CustomQueue.java:147)
at java.util.concurrent.ThreadPoolExecutor.afterExecute(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- <0x00002aaf5f9ed518> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"xxx>Job Read-1" daemon prio=10 tid=0x00002aca00183000 nid=0x6a39 waiting for monitor entry [0x0000000052c42000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.mycompany.collections.CustomQueue.remove(CustomQueue.java:101)
- waiting to lock <0x00002aae6465a650> (a java.util.ArrayDeque)
at com.mycompany.collections.CustomQueue.trim(CustomQueue.java:318)
at com.mycompany.collections.CustomQueue.itemProcessed(CustomQueue.java:302)
at com.mycompany.collections.CustomQueue.trackCompleted(CustomQueue.java:147)
at java.util.concurrent.ThreadPoolExecutor.afterExecute(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- <0x00002aaf5f9ecde8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"xxx>Job Read-0" daemon prio=10 tid=0x0000000006a83000 nid=0x6a36 waiting for monitor entry [0x000000005287f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.mycompany.collections.CustomQueue.remove(CustomQueue.java:101)
- waiting to lock <0x00002aae6465a650> (a java.util.ArrayDeque)
at com.mycompany.collections.CustomQueue.trim(CustomQueue.java:318)
at com.mycompany.collections.CustomQueue.itemProcessed(CustomQueue.java:302)
at com.mycompany.collections.CustomQueue.trackCompleted(CustomQueue.java:147)
at java.util.concurrent.ThreadPoolExecutor.afterExecute(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Here is the Java code extracted, which shows where the error is...
public class Deadlock {
final Deque<Object> delegate = new ArrayDeque<>();
final long maxSize = Long.MAX_VALUE;
private final AtomicLong totalExec = new AtomicLong();
private final Map<Object, AtomicLong> totals = new HashMap<>();
private final Map<Object, Deque<Long>> execTimes = new HashMap<>();
public void trim() {
//Possible optimization is evicting in chunks, segmenting by arrival time
while (this.totalExec.longValue() > this.maxSize) {
final Object t = this.delegate.peek();
final Deque<Long> execTime = this.execTimes.get(t);
final Long exec = execTime.peek();
if (exec != null && this.totalExec.longValue() - exec > this.maxSize) {
//If Job Started Inside of Window, remove and re-loop
remove();
}
else {
//Otherwise exit the loop
break;
}
}
}
public Object remove() {
Object removed;
synchronized (this.delegate) { //4 Threads deadlocking on this line !
removed = this.delegate.pollFirst();
}
if (removed != null) {
itemRemoved(removed);
}
return removed;
}
public void itemRemoved(final Object t) {
//Decrement Total & Queue
final AtomicLong catTotal = this.totals.get(t);
if (catTotal != null) {
if (!this.execTimes.get(t).isEmpty()) {
final Long exec = this.execTimes.get(t).pollFirst();
if (exec != null) {
catTotal.addAndGet(-exec);
this.totalExec.addAndGet(-exec);
}
}
}
}
}

From the documentation for HashMap
Note that this implementation is not synchronized. If multiple threads
access a hash map concurrently, and at least one of the threads
modifies the map structurally, it must be synchronized externally.
(Emphasis theirs)
You are both reading and writing to/from the Maps in an unsynchronized manner.
I see no reason to assume that your code is thread safe.
I suggest that you have an infinite loop in trim caused by this lack of thread safety.
Entering a synchronized block is relatively slow, so it's likely that a thread dump will always show at least a few threads waiting to obtain the lock.

Your first thread is holding the lock while waiting for pollFirst.
"xxx>Job Read-3" daemon prio=10 tid=0x00002aca001a6800 nid=0x6a3b waiting for monitor entry [0x0000000052ec4000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.mycompany.collections.CustomQueue.remove(CustomQueue.java:101)
- locked <0x00002aae6465a650> (a java.util.ArrayDeque)
at com.mycompany.collections.CustomQueue.trim(CustomQueue.java:318)
The other threads are waiting to obtain the lock.
You will need to provide the entire thread dump to determine which thread is holding the lock on 0x0000000052ec4000, which is what is preventing your pollFirst call from returning.

In order to deadlock, you need at least two be locking on at least two objects in the same thread at the same time which is something the code you posted doesn't appear to do. The bug you point to may apply but as I read it, it's a cosmetic issue and that the threads are not 'locked', but waiting to acquire a lock on the object in question (the ArrayDeque). You should see a "deadlock" message in your logs if you have a deadlock. It will call out the two threads that are blocking each other.
I don't believe the thread dump says there are deadlocks. It's simply telling you how many threads are waiting on the monitor at the moment you took the dump. Since only one thread may have the monitor at a given moment, it shouldn't be very surprising.
What behavior are you seeing in your application that lead you to believe you have a deadlock? There's a lot missing from your code particularly where the objects in the delegate Dequeue are coming from. My guess is you don't have an outright deadlock but some other issue that may look like a deadlock.

Thanks to the responses here, it became clear that the issue was none Thread Safe usage of multiple Collections.
To resolve the issue, I've made the trim method synchronized and replaced usage of HashMap with ConcurrentHashMap and ArrayDeque with LinkedBlockingDeque
(Concurrent Collections FTW!)
A further planned enhancement is to change the usage of 2 separate Maps into a single Map containing a Custom Object, that way keeping the operations (in itemRemoved) atomic.

Getting errors leasing tasks

When I try to lease tasks from a pull queue in my application, I get the error below. This didn't happen previously with my code, so something has changed. I suspect that it's a threading issue - these calls are made in a separate thread (created using ThreadManager.createThreadForCurrentRequest) - has that recently been disallowed?
uk.org.jaggard.myapp.BlahBlahBlahRunnable run: Exception leasing tasks. Already tried 0 times.
com.google.apphosting.api.ApiProxy$CancelledException: The API call taskqueue.QueryAndOwnTasks() was explicitly cancelled.
at com.google.apphosting.runtime.ApiProxyImpl.doSyncCall(ApiProxyImpl.java:218)
at com.google.apphosting.runtime.ApiProxyImpl.access$000(ApiProxyImpl.java:68)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:182)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:180)
at java.security.AccessController.doPrivileged(Native Method)
at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:180)
at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:68)
at com.google.appengine.tools.appstats.Recorder.makeSyncCall(Recorder.java:323)
at com.googlecode.objectify.cache.TriggerFutureHook.makeSyncCall(TriggerFutureHook.java:154)
at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:105)
at com.google.appengine.api.taskqueue.QueueApiHelper.makeSyncCall(QueueApiHelper.java:44)
at com.google.appengine.api.taskqueue.QueueImpl.leaseTasksInternal(QueueImpl.java:709)
at com.google.appengine.api.taskqueue.QueueImpl.leaseTasks(QueueImpl.java:731)
at uk.org.jaggard.myapp.BlahBlahBlahRunnable.run(BlahBlahBlahRunnable.java:53)
at com.google.apphosting.runtime.ApiProxyImpl$CurrentRequestThreadFactory$1$1.run(ApiProxyImpl.java:997)
at java.security.AccessController.doPrivileged(Native Method)
at com.google.apphosting.runtime.ApiProxyImpl$CurrentRequestThreadFactory$1.run(ApiProxyImpl.java:994)
at java.lang.Thread.run(Thread.java:679)
at com.google.apphosting.runtime.ApiProxyImpl$CurrentRequestThreadFactory$2$1.run(ApiProxyImpl.java:1031)

I suspect the warnings in your application's logs will contain
"Thread was interrupted, throwing CancelledException."
Java's InterruptedException is discussed in
http://www.ibm.com/developerworks/java/library/j-jtp05236/index.html
and the book "Java Concurrency in Practice".