Develop Reference

java.lang.Thread.State: WAITING (on object monitor)

I am trying to create weblogic domain using python code which got stuck at filehandler and below is code and stacktrace which is waiting on something can you help to fix it ?
import os
import sys
import wl as myWLST
from java.lang import System,Runtime,Process,String
from java.io import BufferedReader,InputStreamReader,File
from java.util.logging import FileHandler,Logger,SimpleFormatter
from com.oracle.cie.domain.script.jython import WLSTException
from com.oracle.cie.domain.script import ScriptException
from WeblogicConfigXmlParser import XmlParser
#=====================
# Program starts here
#=====================
if __name__ == "__main__":
print "in main"
print len(sys.argv)
if len(sys.argv) < 2:
print 'Insufficent arguments'
System.exit(-1)
else:
try:
print "inside try - JASON"
print (os.getcwd())
fileHandler = FileHandler("weblogic.log",1);
print "inside try 2"
fileHandler.setFormatter(SimpleFormatter());`
it is stuck at line "fileHandler = FileHandler("weblogic.log",1);",
Stack trace :
Full thread dump Java HotSpot(TM) Server VM (24.255-b255-jre1.7.0.25.ia-rc1 mixed mode):
"Attach Listener" daemon prio=8 tid=0x00835000 nid=13 lwp_id=3559300 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" daemon prio=8 tid=0x00676c00 nid=12 lwp_id=3559264 runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=8 tid=0x00670600 nid=10 lwp_id=3559262 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=8 tid=0x0066ce00 nid=9 lwp_id=3559261 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=8 tid=0x0066aa00 nid=8 lwp_id=3559260 runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=0x00653a00 nid=7 lwp_id=3559259 in Object.wait() [0x484c0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x73a28788> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:152)
- locked <0x73a28788> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:168)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:248)
"Reference Handler" daemon prio=8 tid=0x00651600 nid=6 lwp_id=3559258 in Object.wait() [0x4a3c0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x73a28818> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:140)
- locked <0x73a28818> (a java.lang.ref.Reference$Lock)
"main" prio=8 tid=0x00473400 nid=1 lwp_id=-1 runnable [0x7fffd000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:90)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1098)
at java.nio.channels.FileChannel.tryLock(FileChannel.java:1154)
at java.util.logging.FileHandler.openFiles(FileHandler.java:438)
- locked <0x710b6228> (a java.util.HashMap)
at java.util.logging.FileHandler.<init>(FileHandler.java:310)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.python.core.PyReflectedConstructor.__call__(Unknown Source)
at org.python.core.PyJavaInstance.__init__(Unknown Source)
at org.python.core.PyJavaClass.__call__(Unknown Source)
at org.python.core.PyObject.__call__(Unknown Source)
at org.python.pycode._pyx0.f$0(/SHARED/scripts/weblogicscripts/WeblogicDomainManager.py:428)
at org.python.pycode._pyx0.call_function(/SHARED/scripts/weblogicscripts/WeblogicDomainManager.py)
at org.python.core.PyTableCode.call(Unknown Source)
at org.python.core.PyCode.call(Unknown Source)
at org.python.core.Py.runCode(Unknown Source)
at org.python.core.__builtin__.execfile_flags(Unknown Source)
at org.python.util.PythonInterpreter.execfile(Unknown Source)
at org.python.util.jython.main(Unknown Source)
"VM Thread" prio=8 tid=0x00640400 nid=5 lwp_id=3559257 runnable
"GC task thread#0 (ParallelGC)" prio=8 tid=0x00495000 nid=3 lwp_id=3559255 runnable
"GC task thread#1 (ParallelGC)" prio=8 tid=0x004bb600 nid=4 lwp_id=3559256 runnable
"VM Periodic Task Thread" prio=8 tid=0x00674800 nid=11 lwp_id=3559263 waiting on condition
JNI global references: 273
Not able to figure why it is stuck at filehandler, it is creating any weblogic.log either and not getting timedout. even i used "-XX:CompileCommand="exclude,java/lang/Object.wait" based on this post object waiting but still no luck.

This is solved after changing the location of the log file from /SHARED to local
OLD :
fileHandler = FileHandler("/SHARED/weblogic.log",1);
print "inside try 2"
fileHandler.setFormatter(SimpleFormatter());`
NEW :
fileHandler = FileHandler("/home/user/weblogic.log",1);
print "inside try 2"
fileHandler.setFormatter(SimpleFormatter());`
filesystem, there is some issue with /shared file system while writing files, which is why it is stuck.

Java Runnable threads stuck in native methods, consuming high CPU for a long time

Hi i have a case where a bunch of same thread are in RUNNABLE state executing a Native method and stuck in it for a long time (more than an Hour)
"http-bio-8080-exec-806" daemon prio=10 tid=0x00007fd4b0174800 nid=0x7e09 runnable [0x00007fd84a141000]
java.lang.Thread.State: RUNNABLE
at java.lang.Class.getSuperclass(Native Method)
"http-bio-8080-exec-795" daemon prio=10 tid=0x00007fce6c007800 nid=0x7e01 runnable [0x00007fd3f037a000]
java.lang.Thread.State: RUNNABLE
at java.lang.Class.getSuperclass(Native Method)
"http-bio-8080-exec-802" daemon prio=10 tid=0x00007fd67c11a000 nid=0x7dff runnable [0x00007fd4723b1000]
java.lang.Thread.State: RUNNABLE
at java.lang.Class.getSuperclass(Native Method)
"http-bio-8080-exec-800" daemon prio=10 tid=0x00007fd67c2a8000 nid=0x7dfe runnable [0x00007fd7f2dfe000]
java.lang.Thread.State: RUNNABLE
at java.lang.Class.getSuperclass(Native Method)
"http-bio-8080-exec-783" daemon prio=10 tid=0x00007fcd68001800 nid=0x7699 runnable [0x00007fd2b05fd000]
java.lang.Thread.State: RUNNABLE
at java.lang.Class.getSuperclass(Native Method)
"http-bio-8080-exec-612" daemon prio=10 tid=0x00007fd53001b000 nid=0x55a9 runnable [0x00007fd5013cf000]
java.lang.Thread.State: RUNNABLE
at java.lang.Class.getSuperclass(Native Method)
at org.apache.commons.lang.builder.EqualsBuilder.reflectionEquals(EqualsBuilder.java:286)
at org.apache.commons.lang.builder.EqualsBuilder.reflectionEquals(EqualsBuilder.java:124)
at com.tngtech.jira.plugins.gadget.data.FakeBarChartEntry.equals(FakeBarChartEntry.java:44)
at java.util.ArrayList.indexOf(Unknown Source)
at java.util.ArrayList.contains(Unknown Source)
at com.tngtech.jira.plugins.gadget.data.BarChartDataSorter.currentIsPriorToBuffer(BarChartDataSorter.java:121)
at com.tngtech.jira.plugins.gadget.data.BarChartDataSorter.sortEntriesAndFillGaps(BarChartDataSorter.java:98)
at com.tngtech.jira.plugins.gadget.data.BarChartDataSorter.sortAndEnrichChartData(BarChartDataSorter.java:27)
at com.tngtech.jira.plugins.gadget.data.BarChartDataCollector.collectChartData(BarChartDataCollector.java:54)
at com.tngtech.jira.plugins.gadget.BarChartGadget.createRestClass(BarChartGadget.java:111)
at com.tngtech.jira.plugins.gadget.BarChartGadget.getVersionsForProject(BarChartGadget.java:85)
at com.tngtech.jira.plugins.gadget.BarChartGadget.getVersionsForProject(BarChartGadget.java:67)
I just can't get whats going on here ? after an hour the threads terminate!

Simple while loop waiting and never starts in Java 1.6 on Xubuntu 12.04

I have a really simple Java code:
public class Main {
public static void main(String[] args) {
int c = 0;
while (c < 10);
{
System.out.println(c);
c++;
}
System.out.println("exit");
}
}
The problem is, the while loop doesn't starts, the program just keeps waiting on the 7th line (at the beginning of the while loop).
I use Oracle Java JDK 1.6u33 on Xubuntu 12.04. On Windows this code works perfectly. Can somebody tell me, what can I do to run this code correctly on Ubuntu?
What I did (and the Thread dump):
gopher#orion:~/Projects/test/src$ javac Main.java
gopher#orion:~/Projects/test/src$ jar cvfm test.jar manifest.txt Main.class
added manifest
adding: Main.class(in = 489) (out= 338)(deflated 30%)
gopher#orion:~/Projects/test/src$ java -jar test.jar
^\2012-08-04 10:26:23
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03 mixed mode):
"Low Memory Detector" daemon prio=10 tid=0x00007fbf080b4800 nid=0xa26 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x00007fbf080b2800 nid=0xa25 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x00007fbf080af800 nid=0xa24 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007fbf080ad800 nid=0xa23 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007fbf08091000 nid=0xa22 in Object.wait() [0x00007fbf0d1f3000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f5b61300> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00000000f5b61300> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x00007fbf0808f000 nid=0xa21 in Object.wait() [0x00007fbf0d2f4000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f5b611d8> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00000000f5b611d8> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x00007fbf08007800 nid=0xa1d runnable [0x00007fbf0eb4b000]
java.lang.Thread.State: RUNNABLE
at Main.main(Main.java:8)
"VM Thread" prio=10 tid=0x00007fbf08088800 nid=0xa20 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fbf0801a800 nid=0xa1e runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fbf0801c000 nid=0xa1f runnable
"VM Periodic Task Thread" prio=10 tid=0x00007fbf080bf000 nid=0xa27 waiting on condition
JNI global references: 887
Heap
PSYoungGen total 9216K, used 317K [0x00000000f5b60000, 0x00000000f65a0000, 0x0000000100000000)
eden space 7936K, 4% used [0x00000000f5b60000,0x00000000f5baf628,0x00000000f6320000)
from space 1280K, 0% used [0x00000000f6460000,0x00000000f6460000,0x00000000f65a0000)
to space 1280K, 0% used [0x00000000f6320000,0x00000000f6320000,0x00000000f6460000)
PSOldGen total 21056K, used 0K [0x00000000e1200000, 0x00000000e2690000, 0x00000000f5b60000)
object space 21056K, 0% used [0x00000000e1200000,0x00000000e1200000,0x00000000e2690000)
PSPermGen total 21248K, used 2774K [0x00000000dc000000, 0x00000000dd4c0000, 0x00000000e1200000)
object space 21248K, 13% used [0x00000000dc000000,0x00000000dc2b5bb8,0x00000000dd4c0000)

You have a ; at the end of the while, remove it and it will work
while (c < 10);
the above will exectue a single line while() and because c is never increased it is an endless loop. change it into
while (c < 10) // no ; at the end

Remove ; from while loop. make it as follows
while (c < 10)

Java thread blocking

i have a problem with my java environement. I'm running Solr 1.3 (search engine) since more then a year now and suddenly i got alot of trouble with it. All my thread pool (250) got randomly blocked once or twice a day. I did not make any change on my solr application or my tomcat server.
I'm running tomcat 5.5.25 and Solr 1.3. I got a thread dump when the system is totally overloaded :
igot like 240 thread like this one :
"http-8080-Processor1" daemon prio=10 tid=0x0000000000b2e000 nid=0x193 waiting for monitor entry [0x000000004066c000..0x000000004066cb20]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.logging.StreamHandler.publish(StreamHandler.java:174)
- waiting to lock <0x00007fe37e72b340> (a java.util.logging.ConsoleHandler)
at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88)
at java.util.logging.Logger.log(Logger.java:472)
at java.util.logging.Logger.doLog(Logger.java:494)
at java.util.logging.Logger.log(Logger.java:517)
at java.util.logging.Logger.info(Logger.java:1036)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1212)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:619)
We can see that this thread is blocked and waiting on : <0x00007fe37e72b340>
The thread who actually own the <0x00007fe37e72b340> is this one :
"http-8080-Processor156" daemon prio=10 tid=0x0000000000df2000 nid=0x1e52 runnable [0x0000000044521000..0x0000000044521c20]
java.lang.Thread.State: RUNNABLE
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
- locked <0x00007fe37e3abcd8> (a java.io.BufferedOutputStream)
at java.io.PrintStream.write(PrintStream.java:430)
- locked <0x00007fe37e3abca0> (a java.io.PrintStream)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
- locked <0x00007fe37e72cd90> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
at java.util.logging.StreamHandler.flush(StreamHandler.java:225)
- locked <0x00007fe37e72b340> (a java.util.logging.ConsoleHandler)
at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:89)
at java.util.logging.Logger.log(Logger.java:472)
at java.util.logging.Logger.doLog(Logger.java:494)
at java.util.logging.Logger.log(Logger.java:517)
at java.util.logging.Logger.info(Logger.java:1036)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1212)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:619)
and this is the last part of my thread dump :
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x00007f6510349800 nid=0xbff waiting on condition [0x0000000041d8d000..0x0000000041d8dd20]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1548)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-1" prio=10 tid=0x0000000000c26400 nid=0xbfe waiting on condition [0x000000004200e000..0x000000004200eca0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f651b275510> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
at java.lang.Thread.run(Thread.java:619)
"Low Memory Detector" daemon prio=10 tid=0x00007f6510004400 nid=0xbfa runnable [0x0000000000000000..0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x00007f6510001000 nid=0xbf9 waiting on condition [0x0000000000000000..0x0000000040d5e340]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x00000000006bc400 nid=0xbf8 waiting on condition [0x0000000000000000..0x0000000040c5d2d0]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00000000006bb000 nid=0xbf7 runnable [0x0000000000000000..0x0000000040b5da30]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x0000000000690c00 nid=0xbf6 in Object.wait() [0x000000004065e000..0x000000004065ed20]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f651aa10258> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x00007f651aa10258> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x000000000068f400 nid=0xbf5 in Object.wait() [0x000000004055d000..0x000000004055dca0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f651aa10338> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00007f651aa10338> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x0000000000622400 nid=0xbeb runnable [0x0000000000000000..0x00007fff69fcbba0]
java.lang.Thread.State: RUNNABLE
"VM Thread" prio=10 tid=0x000000000068a000 nid=0xbf4 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x000000000062cc00 nid=0xbec runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x000000000062e000 nid=0xbed runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x000000000062f400 nid=0xbee runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000000630400 nid=0xbef runnable
"GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000000631800 nid=0xbf0 runnable
"GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000000632c00 nid=0xbf1 runnable
"GC task thread#6 (ParallelGC)" prio=10 tid=0x0000000000634000 nid=0xbf2 runnable
"GC task thread#7 (ParallelGC)" prio=10 tid=0x0000000000635400 nid=0xbf3 runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f6510006800 nid=0xbfb waiting on condition
JNI global references: 1201
I know this is not a thread deadlock prob since one thread is actually runnning with all the ressource every other thread want.
Anyone got an idea of what can cause this prob ?

All your Threads are logging things. They all need to write on the disk from time to time.
Everytime one of your 240 threads hits a logging line there will be disk access issues.
It baffles me that the Thread having the lock is in the RUNNABLE state.
I think it might be waiting for some external ressource to be released (like disk access for example)
Are you running low on disk space? Have you recently changed something in your storage system?

If you are running under Windows and the java application starts a console, be careful not to click in the DOS box. Window's crappy mark and copy "feature" blocks output to the ConsoleHandler. So any logger trying to write to the screen will block. Writing to the Console is done in a native call and so the java thread will appear to be in a RUNNING state when in fact it is blocked, it's just there's no way to feed that blocked status back to the application (because you are in native space).
If the application is blocked (you have clicked in the DOS box) press escape to continue.

I never used java.util.logging, so I don't know whether my suggestion is useful, but netherless:
try to use different instance of java.util.logging.Logger, so not all 240 threads will be blocked on the same monitor
(it will help if different instances of Logger use different instances of java.util.logging.ConsoleHandler) .

It seems that the thread that own "0x00007fe37e72b340" is blocked at the IO level. Maybe a disk (raid?) issue?
can you do a thread dump 5 minutes later the see if the same thread is still blocked?

Flushing after each log record is going to expensive if you have very verbose logs.
A quality fix would be to clean up the logging, probably based around auditing.
As a quick fix, override StreamHandler.flush or OutputStream.flush to not do so immediately. Only flush once every so often. Note, however that you could potentially lose logging data immediately before a crash if you do this.

According to your log, the issue concerns the use of java.util.logging.ConsoleHandler.
First try to disable the console handler by removing it from the 'handlers' and '.handlers' list in '${TOMCAT_HOME}/conf/logging.properties'. See whether the problem still occurs.
If that helps, than it is definitely an issue with the output of the ConsoleHandler. Try checking whether there is are issues concerning the 'catalina.out' file. This is the file to where tomcat redirects its console output.

Java thread dump where main thread has no call stack? (jsvc)

We have a java process running as a daemon (under jsvc). Every several days it just stops doing any work; output to the logfile stops (it is pretty verbose, on 5-minute intervals) and it consumes no CPU or IO.
There are no exceptions logged in the logfile nor in syserr or sysout. The last log statement is just prior to a db commit being done, but there is no open connection on the db server (MySQL) and reviewing the code, there should always be additional log output after that, even if it had encountered an exception that was going to bubble up.
The most curious thing I find is that in the thread dump (included below), there's no thread in our code at all, and the main thread seems to have no context whatsoever:
"main" prio=10 tid=0x0000000000614000 nid=0x445d runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
As noted earlier, this is a daemon process running using jsvc, but I don't know if that has anything to do with it (I can restructure the code to also allow running it directly, to test).
Any suggestions on what might be happening here?
Thanks... dwh
Full thread dump:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed mode):
"MySQL Statement Cancellation Timer" daemon prio=10 tid=0x00002aaaf81b8800 nid=0x447b in Object.wait() [0x00002aaaf6a22000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab5556d50> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
- locked <0x00002aaab5556d50> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Low Memory Detector" daemon prio=10 tid=0x00000000006a4000 nid=0x4479 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x00000000006a1000 nid=0x4477 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x000000000069d000 nid=0x4476 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x000000000069b000 nid=0x4465 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x0000000000678800 nid=0x4464 in Object.wait() [0x00002aaaf61d6000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab54a1cb8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00002aaab54a1cb8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x0000000000676800 nid=0x4463 in Object.wait() [0x00002aaaf60d5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab54a1cf0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00002aaab54a1cf0> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x0000000000614000 nid=0x445d runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"VM Thread" prio=10 tid=0x0000000000670000 nid=0x4462 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x000000000061e000 nid=0x445e runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000000620000 nid=0x445f runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000000622000 nid=0x4460 runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000000623800 nid=0x4461 runnable
"VM Periodic Task Thread" prio=10 tid=0x00000000006a6800 nid=0x447a waiting on condition
JNI global references: 797
Heap
PSYoungGen total 162944K, used 48388K [0x00002aaadff40000, 0x00002aaaf2ab0000, 0x00002aaaf5490000)
eden space 102784K, 47% used [0x00002aaadff40000,0x00002aaae2e81170,0x00002aaae63a0000)
from space 60160K, 0% used [0x00002aaaeb850000,0x00002aaaeb850000,0x00002aaaef310000)
to space 86720K, 0% used [0x00002aaae63a0000,0x00002aaae63a0000,0x00002aaaeb850000)
PSOldGen total 699072K, used 699072K [0x00002aaab5490000, 0x00002aaadff40000, 0x00002aaadff40000)
object space 699072K, 100% used [0x00002aaab5490000,0x00002aaadff40000,0x00002aaadff40000)
PSPermGen total 21248K, used 9252K [0x00002aaab0090000, 0x00002aaab1550000, 0x00002aaab5490000)
object space 21248K, 43% used [0x00002aaab0090000,0x00002aaab09993e8,0x00002aaab1550000)

Not all Throwables are Exceptions. Does your error logging code catch Errors (OutOfMemoryError, StackOverflowError, etc)?

Another couple of possibilities:
The exception might be being thrown on a worker thread that does not log exceptions. You can address this by using Thread.setDefaultUncaughtExceptionHandler(...).
The exception that is being thrown might override the Throwable.fillInStackTrace() method. (This is a long shot ... but apparently some people do this in a misguided attempt to prevent reverse engineering.)