I see this thread in my jstack that does not appear to be moving at all. Any pointers on how to figure out why it's stuck? I don't see any locks or anything, the only suspicious thing is the "Object.wait()" reference.
"main" prio=10 tid=0x00007f3a8000b000 nid=0x942 in Object.wait() [0x00007f3a89539000]
java.lang.Thread.State: RUNNABLE
at org.joda.time.DateTimeZone.<clinit>(DateTimeZone.java:95)
at org.joda.time.format.DateTimeFormatter.withZoneUTC(DateTimeFormatter.java:301)
at com.amazonaws.auth.AWS4Signer.<clinit>(AWS4Signer.java:44)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at java.lang.Class.newInstance0(Class.java:372)
at java.lang.Class.newInstance(Class.java:325)
at com.amazonaws.auth.SignerFactory.createSigner(SignerFactory.java:121)
at com.amazonaws.auth.SignerFactory.lookupAndCreateSigner(SignerFactory.java:107)
at com.amazonaws.auth.SignerFactory.getSigner(SignerFactory.java:80)
at com.amazonaws.AmazonWebServiceClient.computeSignerByServiceRegion(AmazonWebServiceClient.java:311)
at com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:284)
at com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:160)
Also, line 95 in DateTimeZone.java at the top of the stack is this:
public static final DateTimeZone UTC = new FixedDateTimeZone("UTC", "UTC", 0, 0);
There is another thread that's also stuck in a similar place:
"FeatureManagerService" daemon prio=10 tid=0x00007f3a8056a800 nid=0x94f in Object.wait() [0x00007f3a84151000]
java.lang.Thread.State: RUNNABLE
at com.amazonaws.util.DateUtils.<clinit>(DateUtils.java:35)
at com.amazonaws.services.s3.internal.ServiceUtils.<clinit>(ServiceUtils.java:59)
at com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:123)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:348)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:245)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3711)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3664)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:620)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:603)
And DateUtils.java:35 is:
private static final DateTimeZone GMT = new FixedDateTimeZone("GMT", "GMT", 0, 0);
I already tried looking into it with jvisualvm/jhat but didn't really get very far.
Note that this is a live process, not something I am running in my debugger locally and after restart it works fine so it appears to be intermittent.
Any help would be appreciated!
Thanks!
Update using the mixed mode in jstack seems to give some more insight - it's waiting on a pthread_cond_wait:
----------------- 2370 -----------------
0x00007f3a89115414 __pthread_cond_wait + 0xc4
0x00007f3a8833a03c _ZN13ObjectMonitor4waitElbP6Thread + 0x7dc
0x00007f3a88117fbb _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread + 0x36b
0x00007f3a881182ca _ZN13instanceKlass10initializeEP6Thread + 0x6a
0x00007f3a8814d3f3 _ZN18InterpreterRuntime4_newEP10JavaThreadP19constantPoolOopDesci + 0x143
0x00007f3a7d01d9ee * org.joda.time.DateTimeZone.<clinit>() bci:0 line:95 (Interpreted frame)
0x00007f3a7d0004f7 <StubRoutines>
...
Maybe it's not stuck. It's just calling new DateTimeZone() in a loop, and the constructor does some computations. Every time you look at this thread, it's inside DateTimeZone() - but it's a different DateTimeZone() each time.
Which then gets discarded. Happened to me quite a few times.
As found by #naumcho, this proved to be a bug (https://github.com/JodaOrg/joda-time/issues/171).
Based on the information provided (stack traces of two different threads + source line) one could have suspected a deadlock because both threads are trying to instantiate a new object of the same type FixedDateTimeZone.
The next step to confirm that would be to use GDB to inspect the stack frames around __pthread_cond_wait().
Related
Rinning jstack -e produces a dump like this (at least in Java 19):
"Thread-0" #25 [23276] prio=5 os_prio=0 cpu=0.00ms elapsed=593.30s allocated=6720B defined_classes=1 tid=0x0000023dafe60b20 nid=23276 waiting for monitor entry [0x000000796a4ff000]
What does "defined_classes" mean here?
This output is coming from the enhancement JDK-8200720. Its implementation defines this values as follow:
defined_classes=... : The number of classes defined by this thread
This might hint to a thread that loads too many classes.
It was added in commit d1b24f2ceca5 on 25 Jun 2018.
This attribute gives the number of classes defined by this thread.
While parsing a date string the thread hung in the native method. The thread is stopped in the same state for the last 18 hours(Thread dumb in different times shows the same show the same stack for the thread in question).
Please suggest what might be the error
"DateParsingThread - 001" #228 daemon prio=5 os_prio=0 tid=0x00007f98e0029000 nid=0x9c2 runnable [0x00007f98016ee000]
java.lang.Thread.State: RUNNABLE
at java.security.AccessController.doPrivileged(Native Method)
at sun.util.resources.LocaleData.getBundle(LocaleData.java:163)
at sun.util.resources.LocaleData.getDateFormatData(LocaleData.java:127)
at sun.util.locale.provider.LocaleResources.getCalendarNames(LocaleResources.java:325)
at sun.util.locale.provider.CalendarNameProviderImpl.getDisplayNamesImpl(CalendarNameProviderImpl.java:157)
at sun.util.locale.provider.CalendarNameProviderImpl.getDisplayNames(CalendarNameProviderImpl.java:139)
at sun.util.locale.provider.CalendarDataUtility$CalendarFieldValueNamesMapGetter.getObject(CalendarDataUtility.java:178)
at sun.util.locale.provider.CalendarDataUtility$CalendarFieldValueNamesMapGetter.getObject(CalendarDataUtility.java:154)
at sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObjectImpl(LocaleServiceProviderPool.java:281)
at sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObject(LocaleServiceProviderPool.java:265)
at sun.util.locale.provider.CalendarDataUtility.retrieveFieldValueNames(CalendarDataUtility.java:88)
at java.util.Calendar.getDisplayNames(Calendar.java:2178)
at java.text.SimpleDateFormat.getDisplayNamesMap(SimpleDateFormat.java:2366)
at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:1972)
at java.text.SimpleDateFormat.parse(SimpleDateFormat.java
at java.text.DateFormat.parse(DateFormat.java:364)
Any suggestions to find the solutions are welcome.
"pool-11-thread-1" prio=10 tid=0x0a974c00 nid=0x7210 runnable [0x3f3ad000]
java.lang.Thread.State: RUNNABLE
at oracle.jdbc.driver.T2CStatement.t2cDefineExecuteFetch(Native Method)
at oracle.jdbc.driver.T2CPreparedStatement.doDefineExecuteFetch(T2CPreparedStatement.java:878)
at oracle.jdbc.driver.T2CPreparedStatement.executeForRows(T2CPreparedStatement.java:760)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1062)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1126)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3339)
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3445)
- locked <0x69579fb0> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x66157d68> (a oracle.jdbc.driver.T2CConnection)
at org.jboss.resource.adapter.jdbc.CachedPreparedStatement.execute(CachedPreparedStatement.java:216)
at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:209)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:180)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery(GeneralStatement.java:205)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralStatement.java:173)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryForObject(GeneralStatement.java:104)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:561)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:536)
at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForObject(SqlMapSessionImpl.java:93)
at org.springframework.orm.ibatis.SqlMapClientTemplate$1.doInSqlMapClient(SqlMapClientTemplate.java:273)
at org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:209)
at org.springframework.orm.ibatis.SqlMapClientTemplate.queryForObject(SqlMapClientTemplate.java:271)
at com.alipay.bipgw.common.dal.bankchannel.ibatis.IbatisBipBusiOrderDAO.queryOrderOutTime(IbatisBipBusiOrderDAO.java:319)
at sun.reflect.GeneratedMethodAccessor3333.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at com.alipay.bipgw.common.dal.monitor.DalMonitorInterceptor.invoke(DalMonitorInterceptor.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy79.queryOrderOutTime(Unknown Source)
at com.alipay.bipgw.prodcore.repository.impl.BusiOrderRepositoryImpl.queryOrderOutTime(BusiOrderRepositoryImpl.java:402)
at com.alipay.bipgw.prodcore.listener.ProdStatusChangeTimeoutTaskListener.execute(ProdStatusChangeTimeoutTaskListener.java:148)
at com.alipay.bipgw.prodcore.listener.ProdStatusChangeTimeoutTaskListener.access$000(ProdStatusChangeTimeoutTaskListener.java:60)
at com.alipay.bipgw.prodcore.listener.ProdStatusChangeTimeoutTaskListener$1.run(ProdStatusChangeTimeoutTaskListener.java:104)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
the java programm like this :
public void onUniformEvent(UniformEvent message, UniformEventContext uContext) {
try {
// single thread running
service.execute(new Runnable() {
public void run() {
try{
execute();
}catch(Exception e ){
logger.error("working error",e);
}
}
});
} catch (RejectedExecutionException e) {
logger.error("ProdStatusChangeTimeoutTaskListener:error", e);
} catch (Exception e) {
logger.error("ProdStatusChangeTimeoutTaskListener:error", e);
}
}
//omit the body
private void execute() {.....}
and the execute method will not start any thread.
in two days i dump several thread dump
2013-03-04 16:54:12
- locked <0x695f91f0> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x6615a2d0> (a oracle.jdbc.driver.T2CConnection)
2013-03-04 17:20:53
- locked <0x695f91f0> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x6615a2d0> (a oracle.jdbc.driver.T2CConnection)
2013-03-05 10:58:30
- locked <0x6957bec8> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x66157e90> (a oracle.jdbc.driver.T2CConnection)
2013-03-05 17:16:31
- locked <0x69579fb0> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x66157d68> (a oracle.jdbc.driver.T2CConnection)
seems like the lock hold by jdbc client has changed, but the first two in 2013-03-04 16:54:12 and 2013-03-04 17:20:53, they are the same
I am using a Excutors.newSingleThreadExecutor() doing a query job in backgroud, and the following task will be submit to this executor Service in 20 minutes interval, but the work thread seems to hangs while executing the query , so the following task will not be executed. It last for several days , no exception occur and no log output at all, somebody can help me ? thanks
The problem is likely to lie in your actual query - is it a long running task? Do you know how long on average it takes to complete?
Its possible (but this is dependent on the query itself) that a previous query is locking tables which in turn block later ones.
The first thing I would do it to verify that the query does indeed complete within 20 minutes, and does so consistently. After you know that it does you can then investigate other possible causes for the hanging behavior you're seeing.
When the query is running I'd also suggest that you check the explain plan for your query (see what its doing - whats taking the most time etc). I'd also check that the relevant statistics are up to date (perhaps less of an issue on newer versions of Oracle)
This problem has been solved~
after several real time executes , out dba monitor the db server side, and find that the query time is 210s which execeeds the query-out-time(180s), but the cancel request of the stamentent may not be effective and the db server may not response to that request . so the thread which porform the original query hangs. the similar situation see [When I call PreparedStatement.cancel() in a JDBC application, does it actually kill it in an Oracle database?
the solution can been done by following:
add sqlprofile to the target tableļ¼and make sql using skip scan to short the query time.
but the deepest reason why hanging will occur is still unknown.
When I try to lease tasks from a pull queue in my application, I get the error below. This didn't happen previously with my code, so something has changed. I suspect that it's a threading issue - these calls are made in a separate thread (created using ThreadManager.createThreadForCurrentRequest) - has that recently been disallowed?
uk.org.jaggard.myapp.BlahBlahBlahRunnable run: Exception leasing tasks. Already tried 0 times.
com.google.apphosting.api.ApiProxy$CancelledException: The API call taskqueue.QueryAndOwnTasks() was explicitly cancelled.
at com.google.apphosting.runtime.ApiProxyImpl.doSyncCall(ApiProxyImpl.java:218)
at com.google.apphosting.runtime.ApiProxyImpl.access$000(ApiProxyImpl.java:68)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:182)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:180)
at java.security.AccessController.doPrivileged(Native Method)
at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:180)
at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:68)
at com.google.appengine.tools.appstats.Recorder.makeSyncCall(Recorder.java:323)
at com.googlecode.objectify.cache.TriggerFutureHook.makeSyncCall(TriggerFutureHook.java:154)
at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:105)
at com.google.appengine.api.taskqueue.QueueApiHelper.makeSyncCall(QueueApiHelper.java:44)
at com.google.appengine.api.taskqueue.QueueImpl.leaseTasksInternal(QueueImpl.java:709)
at com.google.appengine.api.taskqueue.QueueImpl.leaseTasks(QueueImpl.java:731)
at uk.org.jaggard.myapp.BlahBlahBlahRunnable.run(BlahBlahBlahRunnable.java:53)
at com.google.apphosting.runtime.ApiProxyImpl$CurrentRequestThreadFactory$1$1.run(ApiProxyImpl.java:997)
at java.security.AccessController.doPrivileged(Native Method)
at com.google.apphosting.runtime.ApiProxyImpl$CurrentRequestThreadFactory$1.run(ApiProxyImpl.java:994)
at java.lang.Thread.run(Thread.java:679)
at com.google.apphosting.runtime.ApiProxyImpl$CurrentRequestThreadFactory$2$1.run(ApiProxyImpl.java:1031)
I suspect the warnings in your application's logs will contain
"Thread was interrupted, throwing CancelledException."
Java's InterruptedException is discussed in
http://www.ibm.com/developerworks/java/library/j-jtp05236/index.html
and the book "Java Concurrency in Practice".
I'm right now debugging a program which has two threads per one external process, and those two threads keep on reading Process.getErrorStream() and Process.getInputStream() using a while ((i = in.read(buf, 0, buf.length)) >= 0) loop.
Sometimes when the external process crashes due to a JVM crash (see these hs_err_pid.log files), those threads which read the stdout/stderr of that external process begin consuming 100% CPU and never exit. The loop body is not being executed (I've added a logging statement there), so the infinite loop appears to be inside the native method java.io.FileInputStream.readBytes.
I've reproduced this on both Windows 7 64-bit (jdk1.6.0_30 64-bit, jdk1.7.0_03 64-bit), and Linux 2.6.18 (jdk1.6.0_21 32-bit). The code in question is here and it is used like this. See those links for the full code - here are the interesting bits:
private final byte[] buf = new byte[256];
private final InputStream in;
...
int i;
while ((i = this.in.read(this.buf, 0, this.buf.length)) >= 0) {
...
}
The stack traces look like
"PIT Stream Monitor" daemon prio=6 tid=0x0000000008869800 nid=0x1f70 runnable [0x000000000d7ff000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:220)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0x00000007c89d6d90> (a java.io.BufferedInputStream)
at org.pitest.util.StreamMonitor.readFromStream(StreamMonitor.java:38)
at org.pitest.util.StreamMonitor.process(StreamMonitor.java:32)
at org.pitest.util.AbstractMonitor.run(AbstractMonitor.java:19)
Locked ownable synchronizers:
- None
or
"PIT Stream Monitor" daemon prio=6 tid=0x0000000008873000 nid=0x1cb8 runnable [0x000000000e3ff000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:220)
at org.pitest.util.StreamMonitor.readFromStream(StreamMonitor.java:38)
at org.pitest.util.StreamMonitor.process(StreamMonitor.java:32)
at org.pitest.util.AbstractMonitor.run(AbstractMonitor.java:19)
Locked ownable synchronizers:
- None
With the Sysinternals Process Explorer I was able to get native stack traces of those threads. Most often, over 80% of the time, the stack trace looks like this:
ntdll.dll!NtReadFile+0xa
KERNELBASE.dll!ReadFile+0x7a
kernel32.dll!ReadFile+0x59
java.dll!handleRead+0x2c
java.dll!VerifyClassCodesForMajorVersion+0x1d1
java.dll!Java_java_io_FileInputStream_readBytes+0x1d
This also happens quite often:
ntdll.dll!RtlNtStatusToDosErrorNoTeb+0x52
ntdll.dll!RtlNtStatusToDosError+0x23
KERNELBASE.dll!GetCurrentThreadId+0x2c
KERNELBASE.dll!CreatePipe+0x21a
kernel32.dll!ReadFile+0x59
java.dll!handleRead+0x2c
java.dll!VerifyClassCodesForMajorVersion+0x1d1
java.dll!Java_java_io_FileInputStream_readBytes+0x1d
ntdll.dll!RtlNtStatusToDosErrorNoTeb+0x42
ntdll.dll!RtlNtStatusToDosError+0x23
KERNELBASE.dll!GetCurrentThreadId+0x2c
KERNELBASE.dll!CreatePipe+0x21a
kernel32.dll!ReadFile+0x59
java.dll!handleRead+0x2c
java.dll!VerifyClassCodesForMajorVersion+0x1d1
java.dll!Java_java_io_FileInputStream_readBytes+0x1d
And sometimes it's executing this part of the code:
java.dll!VerifyClassCodesForMajorVersion+0xc3
java.dll!Java_java_io_FileInputStream_readBytes+0x1d
java.dll!Java_sun_io_Win32ErrorMode_setErrorMode+0x847c
java.dll!VerifyClassCodesForMajorVersion+0xd7
java.dll!Java_java_io_FileInputStream_readBytes+0x1d
jvm.dll!JNI_GetCreatedJavaVMs+0x1829f
java.dll!VerifyClassCodesForMajorVersion+0x128
java.dll!Java_java_io_FileInputStream_readBytes+0x1d
jvm.dll+0x88c1
jvm.dll!JNI_GetCreatedJavaVMs+0x182a7
java.dll!VerifyClassCodesForMajorVersion+0x128
java.dll!Java_java_io_FileInputStream_readBytes+0x1d
java.dll!VerifyClassCodesForMajorVersion+0x10b
java.dll!Java_java_io_FileInputStream_readBytes+0x1d
jvm.dll!JNI_CreateJavaVM+0x1423
java.dll!VerifyClassCodesForMajorVersion+0x190
java.dll!Java_java_io_FileInputStream_readBytes+0x1d
jvm.dll+0x88bf
jvm.dll!JNI_CreateJavaVM+0x147d
java.dll!VerifyClassCodesForMajorVersion+0x190
java.dll!Java_java_io_FileInputStream_readBytes+0x1d
java.dll!VerifyClassCodesForMajorVersion+0x1aa
java.dll!Java_java_io_FileInputStream_readBytes+0x1d
java.dll!VerifyClassCodesForMajorVersion+0x1c3
java.dll!Java_java_io_FileInputStream_readBytes+0x1d
java.dll!VerifyClassCodesForMajorVersion+0x224
java.dll!Java_java_io_FileInputStream_readBytes+0x1d
Any ideas how to solve this problem? Is this a known problem with the JVM? Is there a workaround?
I've not yet been able to reproduce this locally, but the two possible workarounds might be
Play around with in.available().
Redirect stout and stderr in the external process to a socket and
read this from the controlling process instead.