Have an issue with #Retryable in the Async context, I have a service call which is returning a SocketTimeOut exception. This I would have expected to retry 3 times, I do have #EnableRetry, however I am seeing something I little strange in the logs, a sleep interruptedException. Here is part of the stack trace.
Caused by: java.lang.InterruptedException: sleep interrupted
at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:118) ~[spring-retry-1.2.1.RELEASE.jar!/:na]
at someservice.somemethod(someservice.java) ~[classes/:na]
2018-01-18 18:59:39.818 INFO 14 --- [lTaskExecutor-1] someExceptionHandler : Thread interrupted while sleeping; nested exception is java.lang.InterruptedException: sleep interrupted
at org.springframework.retry.backoff.ThreadWaitSleeper.sleep(ThreadWaitSleeper.java:30) ~[spring-retry-1.2.1.RELEASE.jar!/:na]
at org.springframework.retry.backoff.StatelessBackOffPolicy.backOff(StatelessBackOffPolicy.java:36) ~[spring-retry-1.2.1.RELEASE.jar!/:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_141]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) ~[spring-aop-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) ~[spring-aop-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
Not sure if this is a red herring here, but this happens after the read timeout occurs, I would have expected it to retry but rather I am seeing this in the logs. I know Spring retry has a default wait of 1 second, I am wondering if its being interrupted thus having an impact on its ability to retry.
T.I.A
Caused by: java.lang.InterruptedException: sleep interrupted at
Simply means that something else in your app is interrupting the thread while it is waiting in the backOff, thus killing the retry scenario.
This is by design...
if (this.logger.isDebugEnabled()) {
this.logger
.debug("Abort retry because interrupted: count="
+ context.getRetryCount());
}
...when you interrupt a thread, you are explicitly telling it to stop what it's doing (if it's doing something interruptible, such as sleeping).
Related
I am making a blocking service call in one of my worker verticles that logged a warning. This was "addressed" by increasing the time limit, but, I am more curious about how to read the naming of the thread in the log line - vert.x-worker-thread-3,5,main. The full log was this -
io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-worker-thread-3,5,main] has been blocked for 64134 ms, time limit is 60000
io.vertx.core.VertxException: Thread blocked
What does the 3,5,main indicate? Is it some kind of trace from the main verticle? Thanks.
ThreadName: vert.x-worker-thread-3
ThreadPriority: 5
Source: main
I have an application that is using future for asynchronous execution.
I set the parameter on get method, that the thread should get killed after 10 seconds, when it does not get the response:
Future<RecordMetadata> meta = producer.send(record, new ProducerCallBack());
RecordMetadata data = meta.get(10, TimeUnit.SECONDS);
But the thread get killed after 60 second:
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1124)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:823)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:760)
at io.khinkali.KafkaProducerClient.main(KafkaProducerClient.java:49)
Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
What am I doing wrong?
From the docs:
The threshold for time to block is determined by max.block.ms after
which it throws a TimeoutException.
Check Kafka Appender config in logback.xml, look for:
<producerConfig>max.block.ms=60000</producerConfig>
I set the parameter on get method, that the thread should get killed after 10 seconds, when it does not get the response:
If we are talking about Future.get(...) there is nothing about it that "kills" the thread at all. To quote from the javadocs, the Future.get(...) method:
Waits if necessary for at most the given time for the computation to complete, and then retrieves its result, if available.
If the get(...) method times out then it will throw TimeoutException but your thread is free to continue to run. If you want to stop the thread running then you'll need to catch TimeoutException and then call meta.cancel(true) but even that doesn't guarantee that the thread will be "killed". That causes the thread to be interrupted which means that certain methods will throw InterruptedException or the thread needs to be checking for Thread.currentThread().isInterrupted().
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
Yeah this timeout has nothing to do with the Future.get(...) timeout.
I build a vehicle routing web application using optaplanner. When I tried to deploy my web application to a tomcat 8 server, and tried to run it from my web browser, it produces a warning in my tomcat log file. The log said something about my web application started a thread and failed to stop it, and probably will create a memory leak.
I have write a destroy method where my ExecutorService object will call shutdown method to make sure every thread it started was terminated. Here is my code :
public class OptimizerService implements IOptimizerService {
private ExecutorService executor;
#Override
public synchronized Boolean startSolving() throws Throwable {
executor = Executors.newFixedThreadPool(2);
...
}
...
// other methods
...
#PreDestroy
public synchronized void destroy() {
executor.shutdown();
}
}
But why I still got those warning in tomcat log?
Here is the tomcat log :
09-Jun-2017 08:25:56.377 WARNING [http-nio-18081-exec-295] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [VehicleRouting] appears to have started a thread named [drools-worker-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Any comment will be appreciated. Thanks and regards.
executor.shutdownNow(); interrupts the threads, but executor.shutdown(); does not. That latter just waits until the tasks are finished, good luck if you have a 2h solver running...
If the Solver detects that its thread is interrupted, it terminates (pretty much the same as a normal Termination), which in turns calls KieSession.dispose(). I presume dispose() takes care of any drools spawned threads.
That's the theory at least :)
When I try to lease tasks from a pull queue in my application, I get the error below. This didn't happen previously with my code, so something has changed. I suspect that it's a threading issue - these calls are made in a separate thread (created using ThreadManager.createThreadForCurrentRequest) - has that recently been disallowed?
uk.org.jaggard.myapp.BlahBlahBlahRunnable run: Exception leasing tasks. Already tried 0 times.
com.google.apphosting.api.ApiProxy$CancelledException: The API call taskqueue.QueryAndOwnTasks() was explicitly cancelled.
at com.google.apphosting.runtime.ApiProxyImpl.doSyncCall(ApiProxyImpl.java:218)
at com.google.apphosting.runtime.ApiProxyImpl.access$000(ApiProxyImpl.java:68)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:182)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:180)
at java.security.AccessController.doPrivileged(Native Method)
at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:180)
at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:68)
at com.google.appengine.tools.appstats.Recorder.makeSyncCall(Recorder.java:323)
at com.googlecode.objectify.cache.TriggerFutureHook.makeSyncCall(TriggerFutureHook.java:154)
at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:105)
at com.google.appengine.api.taskqueue.QueueApiHelper.makeSyncCall(QueueApiHelper.java:44)
at com.google.appengine.api.taskqueue.QueueImpl.leaseTasksInternal(QueueImpl.java:709)
at com.google.appengine.api.taskqueue.QueueImpl.leaseTasks(QueueImpl.java:731)
at uk.org.jaggard.myapp.BlahBlahBlahRunnable.run(BlahBlahBlahRunnable.java:53)
at com.google.apphosting.runtime.ApiProxyImpl$CurrentRequestThreadFactory$1$1.run(ApiProxyImpl.java:997)
at java.security.AccessController.doPrivileged(Native Method)
at com.google.apphosting.runtime.ApiProxyImpl$CurrentRequestThreadFactory$1.run(ApiProxyImpl.java:994)
at java.lang.Thread.run(Thread.java:679)
at com.google.apphosting.runtime.ApiProxyImpl$CurrentRequestThreadFactory$2$1.run(ApiProxyImpl.java:1031)
I suspect the warnings in your application's logs will contain
"Thread was interrupted, throwing CancelledException."
Java's InterruptedException is discussed in
http://www.ibm.com/developerworks/java/library/j-jtp05236/index.html
and the book "Java Concurrency in Practice".
I am new to Multi-threading in JAVA and am facing a problem. I am getting an exception which says that
java.lang.InterruptedException: sleep interrupted
this follows a loss to connection to db. I am clueless regarding what is going wrong there. It looks like after thread is interrupted, they are being re-initialized but are not getting started..
following is my run method
public void run() {
LOG.info("Started");
running = true;
while (running) {
Lock readLock = readWriteLock.readLock();
readLock.lock();
long loopDelay;
try {
loopDelay = executor.execute();
if (loopDelay > 0) {
Thread.sleep(loopDelay);
}
} catch (Exception e) {
LOG.info("Executor Interrupted", e);
break;
} finally {
readLock.unlock();
}
}
LOG.info("Stopped");
}
possible solution: after going through the link suggested by david I feel the problem here is the same mentioned by David i.e "A thread cannot process an interrupt while it's sleeping." so to fix this I should handle the Interrrupted Exception in a better way i.e as suggesteb by david.
Following is my stack trace.. Can Some one please help me in understanding the issue
2012-02-03 10:38:09,260 150696427 [taskDiscoveryCallExecutorThread] INFO (TaskExecutorThread.java:89) - Executor Interrupted
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at com.xyz.abc.backgroundtask.impl.TaskExecutorThread.run(TaskExecutorThread.java:86)
2012-02-03 10:38:09,261 150696428 [taskDiscoveryCallExecutorThread] INFO (TaskExecutorThread.java:95) - Stopped
2012-02-03 10:38:09,261 150696428 [main] INFO (TaskExecutorThread.java:69) - shutdown
2012-02-03 10:38:09,262 150696429 [taskRuleExecutorThread] ERROR (JDBCExceptionReporter.java:101) - Cannot get a connection, general error
2012-02-03 10:38:09,262 150696429 [taskRuleExecutorThread] ERROR (JDBCExceptionReporter.java:101) - Cannot get a connection, general error
2012-02-03 10:38:09,294 150696461 [taskRuleExecutorThread] DEBUG (EventScope.java:107) - Destroy scope for customer c02c5ac7-8dee-42aa-b344-ff7f7f6894f5
2012-02-03 10:38:09,294 150696461 [taskRuleExecutorThread] ERROR (TaskExecutor.java:205) - Catch exception
org.springframework.jdbc.UncategorizedSQLException: Hibernate operation: Cannot open connection; uncategorized SQLException for SQL [???]; SQL state [null]; error code [0]; Cannot get a connection, general error; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, general error
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:83)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
at org.springframework.orm.hibernate3.HibernateAccessor.convertJdbcAccessException(HibernateAccessor.java:424)
at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:410)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:411)
at org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:374)
at org.springframework.orm.hibernate3.HibernateTemplate.findByCriteria(HibernateTemplate.java:1046)
at org.springframework.orm.hibernate3.HibernateTemplate.findByCriteria(HibernateTemplate.java:1039)
at com.xyz.abc.web.dao.RuleDao.loadActive(RuleDao.java:53)
at com.xyz.abc.web.service.RuleServiceImpl.createRulesByTemplate(RuleServiceImpl.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy60.createRulesByTemplate(Unknown Source)
at com.xyz.abc.rules.RulesInvoker.checkIsInitialized(RulesInvoker.java:97)
at com.xyz.abc.rules.RulesInvoker.invokeForEvent(RulesInvoker.java:73)
at com.xyz.abc.mdp.RabbitMqMessageProcessor.processMessage(RabbitMqMessageProcessor.java:53)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:186)
at com.xyz.abc.backgroundtask.impl.TaskExecutor.invokeMethod(TaskExecutor.java:156)
at com.xyz.abc.backgroundtask.impl.TaskExecutor.invokeBean(TaskExecutor.java:139)
at com.xyz.abc.backgroundtask.impl.TaskExecutor.doExecute(TaskExecutor.java:116)
at com.xyz.abc.backgroundtask.impl.TaskExecutor.executeTask(TaskExecutor.java:104)
at com.xyz.abc.backgroundtask.impl.TaskExecutor.access$200(TaskExecutor.java:39)
at com.xyz.abc.backgroundtask.impl.TaskExecutor$1.doInTransaction(TaskExecutor.java:85)
at com.xyz.abc.backgroundtask.impl.TaskExecutor$1.doInTransaction(TaskExecutor.java:79)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
at com.xyz.abc.backgroundtask.impl.TaskExecutor.execute(TaskExecutor.java:79)
at com.xyz.abc.backgroundtask.impl.TaskExecutorThread.run(TaskExecutorThread.java:84)
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, general error
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:118)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1596)
at org.hibernate.loader.Loader.doQuery(Loader.java:717)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:270)
at org.hibernate.loader.Loader.doList(Loader.java:2294)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2172)
at org.hibernate.loader.Loader.list(Loader.java:2167)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:119)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1706)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:347)
at org.springframework.orm.hibernate3.HibernateTemplate$36.doInHibernate(HibernateTemplate.java:1056)
at org.springframework.orm.hibernate3.HibernateTemplate$36.doInHibernate(HibernateTemplate.java:1)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
... 33 more
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1104)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
... 50 more
2012-02-03 10:38:09,399 150696566 [taskRuleExecutorThread] INFO (TaskExecutor.java:93) - org.springframework.transaction.UnexpectedRollbackException: Transaction rolled back because it has been marked as rollback-only
2012-02-03 10:38:09,399 150696566 [taskRuleExecutorThread] INFO (TaskExecutorThread.java:89) - Executor Interrupted
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at com.xyz.abc.backgroundtask.impl.TaskExecutorThread.run(TaskExecutorThread.java:86)
2012-02-03 10:38:09,399 150696566 [taskRuleExecutorThread] INFO (TaskExecutorThread.java:95) - Stopped
2012-02-03 10:38:09,400 150696567 [main] INFO (TaskExecutorThread.java:69) - shutdown
2012-02-03 10:38:09,400 150696567 [taskParatureExecutorThread] INFO (TaskExecutorThread.java:89) - Executor Interrupted
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at com.xyz.abc.backgroundtask.impl.TaskExecutorThread.run(TaskExecutorThread.java:86)
2012-02-03 10:38:09,400 150696567 [taskParatureExecutorThread] INFO
(TaskExecutorThread.java:95) - Stopped
2012-02-03 10:38:09,400 150696567 [main] INFO (TaskExecutorThread.java:69) - shutdown
A thread cannot process an interrupt while it's sleeping. So you need to catch the exception when the thread comes out of sleep and then process the interrupt. The method is explained well in this answer. Usually it looks like this:
try
{
Thread.sleep(whatever);
}
catch (InterruptedException e)
{
Thread.currentThread().interrupt(); // restore interrupted status
}
Are you throwing an interrupted Exception somewhere in your code may be to signal that a Thread should stop? Can you provide the code for this method:
com.xyz.abc.backgroundtask.impl.TaskExecutorThread.run
Let's understand with your code:
while (true) {
// Your logic code
}
What does it do? Nothing, it just spins the CPU endlessly. Can we terminate it? Not in Java. It will only stop when the entire JVM stops, when you hit Ctrl-C. There is no way in Java to terminate a thread unless the thread exits by itself. That’s the principle we have to have in mind, and everything else will just be obvious.
So, how do we stop a thread when we need it to stop?
Here is how it is designed in Java. There is a flag in every thread that we can set from the outside. And the thread may check it occasionally and stop its execution. Voluntarily! Here is how:
Thread loop = new Thread(
new Runnable() {
#Override
public void run() {
while (true) {
if (Thread.interrupted()) {
break;
}
// Your logic code
}
}
}
);
loop.start();