I am new to Multi-threading in JAVA and am facing a problem. I am getting an exception which says that
java.lang.InterruptedException: sleep interrupted
this follows a loss to connection to db. I am clueless regarding what is going wrong there. It looks like after thread is interrupted, they are being re-initialized but are not getting started..
following is my run method
public void run() {
LOG.info("Started");
running = true;
while (running) {
Lock readLock = readWriteLock.readLock();
readLock.lock();
long loopDelay;
try {
loopDelay = executor.execute();
if (loopDelay > 0) {
Thread.sleep(loopDelay);
}
} catch (Exception e) {
LOG.info("Executor Interrupted", e);
break;
} finally {
readLock.unlock();
}
}
LOG.info("Stopped");
}
possible solution: after going through the link suggested by david I feel the problem here is the same mentioned by David i.e "A thread cannot process an interrupt while it's sleeping." so to fix this I should handle the Interrrupted Exception in a better way i.e as suggesteb by david.
Following is my stack trace.. Can Some one please help me in understanding the issue
2012-02-03 10:38:09,260 150696427 [taskDiscoveryCallExecutorThread] INFO (TaskExecutorThread.java:89) - Executor Interrupted
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at com.xyz.abc.backgroundtask.impl.TaskExecutorThread.run(TaskExecutorThread.java:86)
2012-02-03 10:38:09,261 150696428 [taskDiscoveryCallExecutorThread] INFO (TaskExecutorThread.java:95) - Stopped
2012-02-03 10:38:09,261 150696428 [main] INFO (TaskExecutorThread.java:69) - shutdown
2012-02-03 10:38:09,262 150696429 [taskRuleExecutorThread] ERROR (JDBCExceptionReporter.java:101) - Cannot get a connection, general error
2012-02-03 10:38:09,262 150696429 [taskRuleExecutorThread] ERROR (JDBCExceptionReporter.java:101) - Cannot get a connection, general error
2012-02-03 10:38:09,294 150696461 [taskRuleExecutorThread] DEBUG (EventScope.java:107) - Destroy scope for customer c02c5ac7-8dee-42aa-b344-ff7f7f6894f5
2012-02-03 10:38:09,294 150696461 [taskRuleExecutorThread] ERROR (TaskExecutor.java:205) - Catch exception
org.springframework.jdbc.UncategorizedSQLException: Hibernate operation: Cannot open connection; uncategorized SQLException for SQL [???]; SQL state [null]; error code [0]; Cannot get a connection, general error; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, general error
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:83)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
at org.springframework.orm.hibernate3.HibernateAccessor.convertJdbcAccessException(HibernateAccessor.java:424)
at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:410)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:411)
at org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:374)
at org.springframework.orm.hibernate3.HibernateTemplate.findByCriteria(HibernateTemplate.java:1046)
at org.springframework.orm.hibernate3.HibernateTemplate.findByCriteria(HibernateTemplate.java:1039)
at com.xyz.abc.web.dao.RuleDao.loadActive(RuleDao.java:53)
at com.xyz.abc.web.service.RuleServiceImpl.createRulesByTemplate(RuleServiceImpl.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy60.createRulesByTemplate(Unknown Source)
at com.xyz.abc.rules.RulesInvoker.checkIsInitialized(RulesInvoker.java:97)
at com.xyz.abc.rules.RulesInvoker.invokeForEvent(RulesInvoker.java:73)
at com.xyz.abc.mdp.RabbitMqMessageProcessor.processMessage(RabbitMqMessageProcessor.java:53)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:186)
at com.xyz.abc.backgroundtask.impl.TaskExecutor.invokeMethod(TaskExecutor.java:156)
at com.xyz.abc.backgroundtask.impl.TaskExecutor.invokeBean(TaskExecutor.java:139)
at com.xyz.abc.backgroundtask.impl.TaskExecutor.doExecute(TaskExecutor.java:116)
at com.xyz.abc.backgroundtask.impl.TaskExecutor.executeTask(TaskExecutor.java:104)
at com.xyz.abc.backgroundtask.impl.TaskExecutor.access$200(TaskExecutor.java:39)
at com.xyz.abc.backgroundtask.impl.TaskExecutor$1.doInTransaction(TaskExecutor.java:85)
at com.xyz.abc.backgroundtask.impl.TaskExecutor$1.doInTransaction(TaskExecutor.java:79)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
at com.xyz.abc.backgroundtask.impl.TaskExecutor.execute(TaskExecutor.java:79)
at com.xyz.abc.backgroundtask.impl.TaskExecutorThread.run(TaskExecutorThread.java:84)
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, general error
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:118)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1596)
at org.hibernate.loader.Loader.doQuery(Loader.java:717)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:270)
at org.hibernate.loader.Loader.doList(Loader.java:2294)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2172)
at org.hibernate.loader.Loader.list(Loader.java:2167)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:119)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1706)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:347)
at org.springframework.orm.hibernate3.HibernateTemplate$36.doInHibernate(HibernateTemplate.java:1056)
at org.springframework.orm.hibernate3.HibernateTemplate$36.doInHibernate(HibernateTemplate.java:1)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
... 33 more
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1104)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
... 50 more
2012-02-03 10:38:09,399 150696566 [taskRuleExecutorThread] INFO (TaskExecutor.java:93) - org.springframework.transaction.UnexpectedRollbackException: Transaction rolled back because it has been marked as rollback-only
2012-02-03 10:38:09,399 150696566 [taskRuleExecutorThread] INFO (TaskExecutorThread.java:89) - Executor Interrupted
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at com.xyz.abc.backgroundtask.impl.TaskExecutorThread.run(TaskExecutorThread.java:86)
2012-02-03 10:38:09,399 150696566 [taskRuleExecutorThread] INFO (TaskExecutorThread.java:95) - Stopped
2012-02-03 10:38:09,400 150696567 [main] INFO (TaskExecutorThread.java:69) - shutdown
2012-02-03 10:38:09,400 150696567 [taskParatureExecutorThread] INFO (TaskExecutorThread.java:89) - Executor Interrupted
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at com.xyz.abc.backgroundtask.impl.TaskExecutorThread.run(TaskExecutorThread.java:86)
2012-02-03 10:38:09,400 150696567 [taskParatureExecutorThread] INFO
(TaskExecutorThread.java:95) - Stopped
2012-02-03 10:38:09,400 150696567 [main] INFO (TaskExecutorThread.java:69) - shutdown
A thread cannot process an interrupt while it's sleeping. So you need to catch the exception when the thread comes out of sleep and then process the interrupt. The method is explained well in this answer. Usually it looks like this:
try
{
Thread.sleep(whatever);
}
catch (InterruptedException e)
{
Thread.currentThread().interrupt(); // restore interrupted status
}
Are you throwing an interrupted Exception somewhere in your code may be to signal that a Thread should stop? Can you provide the code for this method:
com.xyz.abc.backgroundtask.impl.TaskExecutorThread.run
Let's understand with your code:
while (true) {
// Your logic code
}
What does it do? Nothing, it just spins the CPU endlessly. Can we terminate it? Not in Java. It will only stop when the entire JVM stops, when you hit Ctrl-C. There is no way in Java to terminate a thread unless the thread exits by itself. That’s the principle we have to have in mind, and everything else will just be obvious.
So, how do we stop a thread when we need it to stop?
Here is how it is designed in Java. There is a flag in every thread that we can set from the outside. And the thread may check it occasionally and stop its execution. Voluntarily! Here is how:
Thread loop = new Thread(
new Runnable() {
#Override
public void run() {
while (true) {
if (Thread.interrupted()) {
break;
}
// Your logic code
}
}
}
);
loop.start();
Related
We have deployed an Ignite 2.11.1 cluster in Kubernetes. When a client node attempts to join the grid, we are getting the tcp-disco-msg-worker blocked below.
We have used the same configuration successfully with another deployment so we are not certain why we are getting this error. The only change is that in the deployment that is failing we have added resource limits to the configuration.
[20:07:24,201][SEVERE][tcp-disco-msg-worker-[crd]-#2-#48][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [workerName=db-checkpoint-thread, threadName=db-checkpoint-thread-#78, blockedFor=4562s]
[20:07:24] Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=db-checkpoint-thread, igniteInstanceName=null, finished=false, heartbeatTs=1657047082073]]]
[20:07:27,073][SEVERE][tcp-disco-msg-worker-[crd]-#2-#48][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [workerName=sys-stripe-0, threadName=sys-stripe-0-#1, blockedFor=4072s]
Thread [name="qtp2015455415-61", id=61, state=TIMED_WAITING, blockCnt=1, waitCnt=702]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#78322a4, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:973)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1023)
at java.lang.Thread.run(Thread.java:748)
Thread [name="qtp2015455415-60", id=60, state=RUNNABLE, blockCnt=3, waitCnt=679]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked sun.nio.ch.Util$3#7a7b4390
- locked java.util.Collections$UnmodifiableSet#220c68db
- locked sun.nio.ch.EPollSelectorImpl#4f569077
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:183)
at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:190)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.lang.Thread.run(Thread.java:748)
Thread [name="qtp2015455415-59", id=59, state=TIMED_WAITING, blockCnt=1, waitCnt=684]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#78322a4, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:973)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1023)
at java.lang.Thread.run(Thread.java:748)
Thread [name="qtp2015455415-58", id=58, state=TIMED_WAITING, blockCnt=1, waitCnt=688]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#78322a4, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:973)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1023)
at java.lang.Thread.run(Thread.java:748)
Client side errors:
06-Jul-2022 15:16:04.772 INFO [exchange-worker-#45%igniteClientInstance%] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [java.lang.OutOfMemoryError]. The
following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.lang.OutOfMemoryError]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1427)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1415)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1254)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1215)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3219)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.base/java.lang.Thread.run(Thread.java:833)
06-Jul-2022 15:16:43.252 INFO [page-lock-tracker-timeout] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [org.apache.ignite.internal.processors.cache.persi
stence.diagnostic.pagelocktracker.SharedPageLockTracker$State]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.SharedPageLockTracker$State]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1427)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1415)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1254)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1215)
at org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.SharedPageLockTracker.getThreadOperationState(SharedPageLockTracker.java:285)
at org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.SharedPageLockTracker.hangThreads(SharedPageLockTracker.java:301)
at org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.SharedPageLockTracker.access$200(SharedPageLockTracker.java:53)
at org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.SharedPageLockTracker$TimeOutWorker.iteration(SharedPageLockTracker.java:359)
at org.apache.ignite.internal.util.worker.CycleThread.run(CycleThread.java:49)
06-Jul-2022 15:25:42.469 INFO [tcp-comm-worker-#1%igniteClientInstance%-#28%igniteClientInstance%] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [java.lan
g.OutOfMemoryError]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.lang.OutOfMemoryError]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1427)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1415)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1254)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1215)
at org.apache.ignite.spi.communication.tcp.internal.CommunicationWorker.body(CommunicationWorker.java:194)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$6.body(TcpCommuicationSpi.java:928)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
Have an issue with #Retryable in the Async context, I have a service call which is returning a SocketTimeOut exception. This I would have expected to retry 3 times, I do have #EnableRetry, however I am seeing something I little strange in the logs, a sleep interruptedException. Here is part of the stack trace.
Caused by: java.lang.InterruptedException: sleep interrupted
at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:118) ~[spring-retry-1.2.1.RELEASE.jar!/:na]
at someservice.somemethod(someservice.java) ~[classes/:na]
2018-01-18 18:59:39.818 INFO 14 --- [lTaskExecutor-1] someExceptionHandler : Thread interrupted while sleeping; nested exception is java.lang.InterruptedException: sleep interrupted
at org.springframework.retry.backoff.ThreadWaitSleeper.sleep(ThreadWaitSleeper.java:30) ~[spring-retry-1.2.1.RELEASE.jar!/:na]
at org.springframework.retry.backoff.StatelessBackOffPolicy.backOff(StatelessBackOffPolicy.java:36) ~[spring-retry-1.2.1.RELEASE.jar!/:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_141]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) ~[spring-aop-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) ~[spring-aop-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
Not sure if this is a red herring here, but this happens after the read timeout occurs, I would have expected it to retry but rather I am seeing this in the logs. I know Spring retry has a default wait of 1 second, I am wondering if its being interrupted thus having an impact on its ability to retry.
T.I.A
Caused by: java.lang.InterruptedException: sleep interrupted at
Simply means that something else in your app is interrupting the thread while it is waiting in the backOff, thus killing the retry scenario.
This is by design...
if (this.logger.isDebugEnabled()) {
this.logger
.debug("Abort retry because interrupted: count="
+ context.getRetryCount());
}
...when you interrupt a thread, you are explicitly telling it to stop what it's doing (if it's doing something interruptible, such as sleeping).
I am using hosebird client in java to get real time tweets using using twitter streaming api.
After few hours while running it giving below exception.
18-Jul-2017 15:35:50.034 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [hosebird-client-io-thread-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
java.net.SocketInputStream.read(SocketInputStream.java:171)
java.net.SocketInputStream.read(SocketInputStream.java:141)
sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
sun.security.ssl.InputRecord.read(InputRecord.java:503)
sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:251)
org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:209)
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:171)
org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:138)
java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:238)
java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
com.twitter.hbc.common.DelimitedStreamReader.readLine(DelimitedStreamReader.java:72)
com.twitter.hbc.common.DelimitedStreamReader.readLine(DelimitedStreamReader.java:56)
com.twitter.hbc.core.processor.StringDelimitedProcessor.processNextMessage(StringDelimitedProcessor.java:53)
com.twitter.hbc.core.processor.StringDelimitedProcessor.processNextMessage(StringDelimitedProcessor.java:26)
com.twitter.hbc.core.processor.AbstractProcessor.process(AbstractProcessor.java:46)
com.twitter.hbc.httpclient.Connection.processResponse(Connection.java:51)
com.twitter.hbc.httpclient.ClientBase.processConnectionData(ClientBase.java:244)
com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:144)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
I tried to change the hosebird-client-thread to daemon thread as mentioned here, still no luck.
public static Thread getThreadByName(String threadName) {
for (Thread t : Thread.getAllStackTraces().keySet()) {
System.out.println("Thread is: "+ t.getName());
if (t.getName().contains(threadName)) return t;
}
return null;
}
getThreadByName("hosebird-client-io-thread").setDaemon(true);
Any lead would greatly appreciated!
Fixed the memory leak for the thread by stopping it when it doesn't required any more and restart again.
if(hosebirdClient.isDone())
{
hosebirdClient.stop();
updateSystmeParameters(0, getCurrentTimeStamp(), "NA");
Search.startFirehoseFilter(); // this function restart the thread
}
"pool-11-thread-1" prio=10 tid=0x0a974c00 nid=0x7210 runnable [0x3f3ad000]
java.lang.Thread.State: RUNNABLE
at oracle.jdbc.driver.T2CStatement.t2cDefineExecuteFetch(Native Method)
at oracle.jdbc.driver.T2CPreparedStatement.doDefineExecuteFetch(T2CPreparedStatement.java:878)
at oracle.jdbc.driver.T2CPreparedStatement.executeForRows(T2CPreparedStatement.java:760)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1062)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1126)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3339)
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3445)
- locked <0x69579fb0> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x66157d68> (a oracle.jdbc.driver.T2CConnection)
at org.jboss.resource.adapter.jdbc.CachedPreparedStatement.execute(CachedPreparedStatement.java:216)
at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:209)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:180)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery(GeneralStatement.java:205)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralStatement.java:173)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryForObject(GeneralStatement.java:104)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:561)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:536)
at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForObject(SqlMapSessionImpl.java:93)
at org.springframework.orm.ibatis.SqlMapClientTemplate$1.doInSqlMapClient(SqlMapClientTemplate.java:273)
at org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:209)
at org.springframework.orm.ibatis.SqlMapClientTemplate.queryForObject(SqlMapClientTemplate.java:271)
at com.alipay.bipgw.common.dal.bankchannel.ibatis.IbatisBipBusiOrderDAO.queryOrderOutTime(IbatisBipBusiOrderDAO.java:319)
at sun.reflect.GeneratedMethodAccessor3333.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at com.alipay.bipgw.common.dal.monitor.DalMonitorInterceptor.invoke(DalMonitorInterceptor.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy79.queryOrderOutTime(Unknown Source)
at com.alipay.bipgw.prodcore.repository.impl.BusiOrderRepositoryImpl.queryOrderOutTime(BusiOrderRepositoryImpl.java:402)
at com.alipay.bipgw.prodcore.listener.ProdStatusChangeTimeoutTaskListener.execute(ProdStatusChangeTimeoutTaskListener.java:148)
at com.alipay.bipgw.prodcore.listener.ProdStatusChangeTimeoutTaskListener.access$000(ProdStatusChangeTimeoutTaskListener.java:60)
at com.alipay.bipgw.prodcore.listener.ProdStatusChangeTimeoutTaskListener$1.run(ProdStatusChangeTimeoutTaskListener.java:104)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
the java programm like this :
public void onUniformEvent(UniformEvent message, UniformEventContext uContext) {
try {
// single thread running
service.execute(new Runnable() {
public void run() {
try{
execute();
}catch(Exception e ){
logger.error("working error",e);
}
}
});
} catch (RejectedExecutionException e) {
logger.error("ProdStatusChangeTimeoutTaskListener:error", e);
} catch (Exception e) {
logger.error("ProdStatusChangeTimeoutTaskListener:error", e);
}
}
//omit the body
private void execute() {.....}
and the execute method will not start any thread.
in two days i dump several thread dump
2013-03-04 16:54:12
- locked <0x695f91f0> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x6615a2d0> (a oracle.jdbc.driver.T2CConnection)
2013-03-04 17:20:53
- locked <0x695f91f0> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x6615a2d0> (a oracle.jdbc.driver.T2CConnection)
2013-03-05 10:58:30
- locked <0x6957bec8> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x66157e90> (a oracle.jdbc.driver.T2CConnection)
2013-03-05 17:16:31
- locked <0x69579fb0> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x66157d68> (a oracle.jdbc.driver.T2CConnection)
seems like the lock hold by jdbc client has changed, but the first two in 2013-03-04 16:54:12 and 2013-03-04 17:20:53, they are the same
I am using a Excutors.newSingleThreadExecutor() doing a query job in backgroud, and the following task will be submit to this executor Service in 20 minutes interval, but the work thread seems to hangs while executing the query , so the following task will not be executed. It last for several days , no exception occur and no log output at all, somebody can help me ? thanks
The problem is likely to lie in your actual query - is it a long running task? Do you know how long on average it takes to complete?
Its possible (but this is dependent on the query itself) that a previous query is locking tables which in turn block later ones.
The first thing I would do it to verify that the query does indeed complete within 20 minutes, and does so consistently. After you know that it does you can then investigate other possible causes for the hanging behavior you're seeing.
When the query is running I'd also suggest that you check the explain plan for your query (see what its doing - whats taking the most time etc). I'd also check that the relevant statistics are up to date (perhaps less of an issue on newer versions of Oracle)
This problem has been solved~
after several real time executes , out dba monitor the db server side, and find that the query time is 210s which execeeds the query-out-time(180s), but the cancel request of the stamentent may not be effective and the db server may not response to that request . so the thread which porform the original query hangs. the similar situation see [When I call PreparedStatement.cancel() in a JDBC application, does it actually kill it in an Oracle database?
the solution can been done by following:
add sqlprofile to the target table，and make sql using skip scan to short the query time.
but the deepest reason why hanging will occur is still unknown.
I have a java jar file which I am executing from the windows command prompt. The code finishes normally (i.e. does what it is supposed to) but the java process continues to run. The application is single threaded. I need to hit Ctrl-c to stop the process after the code completes to get the command prompt back.
I assume that I could put a System.exit(0) at the end of my main method which would presumably fix this, but I was under the impression that it wasn't necessary. Under what conditions are java processes kept alive at the end of execution? Here's the shell of my main method:
public static void main(String[] args) {
try {
//application code here
Logger.log("Now finished");
} catch (SomeExceptoin e) {
Logger.error("Some error occured");
}
}
Where the logger is my own incredibly basic static class which just dumps messages to System.out.println(). "Now finished" appears in the console, but the process continues to run. Any ideas?
EDIT: As requested, here's the logger code in all its glory (I did warn you :)
public class Logger {
public static void logInfo(String logMessage)
{
System.out.println(timestamp() + logMessage);
}
private static String timestamp()
{
SimpleDateFormat formatter = new SimpleDateFormat("yyyy.MM.dd hh:mm:ss");
String timestamp = "[" + formatter.format(new Date()) + "] ";
return timestamp;
}
}
EDIT EDIT: I put a finally block onto my try above with the thread dump code from the link in instanceofTom's answer. Here's the output:
...
[2010.11.18 11:22:57] Output complete. All processing now finished.
Thread name: Reference Handler
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread name: Finalizer
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread name: Signal Dispatcher
Thread name: Attach Listener
Thread name: Java2D Disposer
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
sun.java2d.Disposer.run(Disposer.java:125)
java.lang.Thread.run(Thread.java:619)
Thread name: main
java.lang.Thread.getStackTrace(Thread.java:1436)
com.my.code.WorkloadManager.visit(WorkloadManager.java:124)
com.my.code.WorkloadManager.visit(WorkloadManager.java:138)
com.my.code.WorkloadManager.main(WorkloadManager.java:71)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:56)
Thread name: com.google.inject.internal.Finalizer
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
com.google.inject.internal.Finalizer.run(Finalizer.java:114)
Thread name: AWT-Windows
sun.awt.windows.WToolkit.eventLoop(Native Method)
sun.awt.windows.WToolkit.run(WToolkit.java:291)
java.lang.Thread.run(Thread.java:619)
Thread name: EventQueueMonitor-ComponentEvtDispatch
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
com.sun.java.accessibility.util.ComponentEvtDispatchThread.run(EventQueueMonitor.java:616) ****************
Thread name: Reference Handler
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread name: Finalizer
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread name: Signal Dispatcher
Thread name: Attach Listener
Thread name: Java2D Disposer
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
sun.java2d.Disposer.run(Disposer.java:125)
java.lang.Thread.run(Thread.java:619)
Thread name: main
java.lang.Thread.getStackTrace(Thread.java:1436)
com.my.code.WorkloadManager.visit(WorkloadManager.java:124)
com.my.code.WorkloadManager.visit(WorkloadManager.java:138)
com.my.code.WorkloadManager.main(WorkloadManager.java:71)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:56)
Thread name: com.google.inject.internal.Finalizer
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
com.google.inject.internal.Finalizer.run(Finalizer.java:114)
Thread name: AWT-Windows
sun.awt.windows.WToolkit.eventLoop(Native Method)
sun.awt.windows.WToolkit.run(WToolkit.java:291)
java.lang.Thread.run(Thread.java:619)
Thread name: EventQueueMonitor-ComponentEvtDispatch
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
com.sun.java.accessibility.util.ComponentEvtDispatchThread.run(EventQueueMonitor.java:616)
Java applications with a UI start event handling threads that are not "daemon" threads. That is, the program will not terminate as long as these threads are running—even if no windows are currently displayed.
Do you have an "Exit" command on your menu? Its Action should invoke System.exit(0).
If everything in your code seems normal it is likely that there is an orphaned thread still running.
The code here describes how to list all of the threads currently running in the JVM
Alternatively you could use a profiler or IDE with built-in profiler to view the running threads.
It would help in answering this question if you could let us know what threads are still running when your code is finished; Even though your code may be single threaded there will likely still be other threads running in the JVM such as the garbage collection thread ( Although GC threads wont stop the JVM from exiting )
A Java process will remain alive so long as it has one or more non-daemon threads running.
Consider using JVisualVM located in the Java JDK's bin directory and attach it to your Java program. There you will be able to perform an analysis of active Threads and their state as well as perform a thread dump.
Based on your thread dump, it looks like one of the Google libraries (perhaps Guice or Guava) is being loaded.
Thread name: com.google.inject.internal.Finalizer
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
com.google.inject.internal.Finalizer.run(Finalizer.java:114)
If so, then this bug could be causing the issue:
http://code.google.com/p/guava-libraries/issues/detail?id=92
It is related to the finalizer thread not being let go correctly. There are various workarounds posted.