Develop Reference

Java Multithreading: Informix 12.10 - Could not do a physical-order read to fetch next row

We are using informix 12.10 version. We are deleting multiple rows of records across 54 tables from Java batch. we are using callable strategy in Multi-threading.
Please refer to the below code:
SampleImpl.java:
Callable<Integer> callable=null;
List<Callable<Integer>> taskList = null;
List<Future<Integer>> futureList = null;
for (Map.Entry<String, String> entry : datas.entrySet()){
callable = new Callable<Integer>(){
public Integer call() throws Exception {
return sampleDel.callSqlDelete();
}
};
taskList.add(callable);
}
SampleDaoImpl:
public void callSqlDelete(){
Statement stmt = null;
connection.setAutoCommit(false);
stmt = connection.createStatement();
stmt.execute("SET LOCK MODE TO WAIT");
stmt.addBatch("DELETE FROM TABLE1 WHERE col1 IN(select from tableAAA where id=101)");
stmt.addBatch("DELETE FROM TABLE2 WHERE col1 IN(select from tableAAA where id=101)"");
int delCnt[] = stmt.executeBatch();
connection.commit();
}
In our java code we have already set lock mode to wait to infinite time interval but still we are getting the below exception:
java.sql.BatchUpdateException: Could not do a physical-order read to fetch next row.
at com.informix.jdbc.IfxStatement.executeBatch(IfxStatement.java:1650)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at oracle.ucp.jdbc.proxy.StatementProxyFactory.invoke(StatementProxyFactory.java:272)
at com.sun.proxy.$Proxy1.executeBatch(Unknown Source)
at com.sample.samplereport.dao.impl.SampleDAOPurgeImpl.processDelByStmts(SampleDAOPurgeImpl.java:1305)
at com.sample.samplereport.util.SamplePlSqlDeleter.callSqlDelete(SamplePlSqlDeleter.java:58)
at com.sample.samplereport.dao.impl.SampleDAOPurgeImpl$1.call(SampleDAOPurgeImpl.java:298)
at com.sample.samplereport.dao.impl.SampleDAOPurgeImpl$1.call(SampleDAOPurgeImpl.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Please help on the above issue?

For this type of error it is usually helpful to see the ISAM error code that the Informix engine also provides. This gives more information on why the operation failed, in this case why it was unable to read the next row. One way to get the ISAM error is to set the environment variable APPENDISAM in the client Java environment. There may well be other ways to achieve this as well. FYI you can find further information in the Informix JDBC Driver documentation at https://www.ibm.com/support/knowledgecenter/SSGU8G_12.1.0/com.ibm.jdbc_pg.doc/ids_jdbc_040.htm
For this problem I suspect the ISAM error may be 143 "deadlock detected." This results when one thread needs to wait on a lock that is held by another thread which in turn is waiting on a lock already held by the first thread. Since you have set lock mode to wait without a timeout the result would be the threads waiting forever so the server returns a deadlock error instead.
To help avoid the problem you should check that row level locking is used in preference to page level locking for TABLE1 and TABLE2. You may also want to check the isolation level used. If using Repeatable Read isolation or the database is mode ANSI then the select statement used in the sub-query will place a lock on every row it considers although these should be minimized if there is an index on the "id" column.
At an application code level deadlock is frequently handled by rolling back the transaction and repeating it.

Find a particular exception and corresponding message

My web application runs on Spring (MVC) 4.2.9.RELEASE, Hibernate 5.1.3.Final, Spring Data 1.8.2.RELEASE, and MS SQL Server (2014).
In the Spring context, I have the following exceptioin hanlder:
<bean id="simpleMappingExceptionResolver" class="myproject.CustomMappingExceptionResolver">
...
</bean>
to catch and save stack trace. I am able to see the following deep in a long stack trace printed in the logs:
......
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
... 113 more
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 73) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:258)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1535)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:467)
How can I find the following exception class in the above exceptioin hanlder (and given an Exception instance) :
com.microsoft.sqlserver.jdbc.SQLServerException
AND the corresponding message:
Transaction (Process ID 73) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

If I understood you correctly you need to catch a nested exception. That's a bit tricky, but doable. You need to have a try-catch block for the top level exception that you expect. In the catch clause, you can use exception.getCause() to step down one nesting level at a time, and see if that level is an instanceof your SQL exception class. You can also check the message if necessary by using getMessage(). If the exception fits your criteria, congratulations you caught it. If not, simply throw it again.
Two things to keep in mind:
this approach may lead to poor performance if many exceptions occur, and only a small fraction of those actually matches your criteria.
if an exception has no cause, then e.getCause() will return e itself. Watch out for infinite recursion here.

Getting a "java.sql.SQLException: You can't operate on a closed Statement", with an open connection

I have a program that makes several requests to a database, opening and closing connections to do what it needs to do. For each connection, it does a select that return 50 results and an update; it does this roughly 10 times per connection. After that, the connection is closed, and a new one is taken. But recently we have been having some random issues in which this SQL exception appears:
java.sql.SQLException: You can't operate on a closed Statement!!!
This error appears randomly. It first appeard mid-execution, and the only moment I managed to reproduce it, it happened at the initiation of the program (when I started it again after that, without making any change, it worked perfectly fine). I've looked around in the code but there's no chance that the connection closes before it can be used (the error occurs while inserting a parameter in a prepared statement). I'm already using c3p0 to manage the connection pool, so I don't know where else to look.
Someone has faced this error before? Any suggestions on where to look or how to reproduce it so I can test it properly?
Edit: Here is the problematic piece of code
try{
//send row to producer
producer.processItem(fields);
if (stmtLasProcessedTransaction == null) {
stmtLasProcessedTransaction = getDbConnection().prepareStatement("UPDATE JTICKET_SUBSCRIBER SET LAST_PROCESSED_ROW = ? WHERE NAME = ? ");
logger.trace("creating statement");
}
//update last processed transaction
logger.trace("Setting the primary key to the prepared statement");
stmtLasProcessedTransaction.setString(1, primaryKey);
logger.trace("Setting the name to the prepared statement");
stmtLasProcessedTransaction.setString(2, name);
logger.trace("Attempting to execute the update on JTICKET_SUBSCRIBER in consumer {}",this.name);
stmtLasProcessedTransaction.executeUpdate();
logger.trace("Commiting execution");
getDbConnection().commit();
logger.trace("Update on JTICKET_SUBSCRIBER in consumer {} executed successfully",this.name);
if (processedRows % 500 == 0) {
logger.trace("resetting prepared statement");
stmtLasProcessedTransaction.close();
logger.trace("statement closed");
stmtLasProcessedTransaction = null;
}
processedRows++;
}catch(SQLException sqlException){
logger.error("An SQL error ocurred while processing consumed item. Closing database connection and statement",sqlException);
try{
stmtLasProcessedTransaction.close();
logger.info("Previous prepared statement of db consumer {} closed",this.name);
}catch(Throwable throwable){
logger.info("Couldn't properly close the prepared statement of db consumer {}",this.name);
}finally{
stmtLasProcessedTransaction=null;
}
try{
databaseConnection.rollback();
logger.info("Rollback of db connection of consumer {} done successfully",this.name);
databaseConnection.close();
logger.info("Previous connection of db consumer {} closed",this.name);
}catch(Throwable throwable){
logger.info("Couldn't rollback and/or close the connection of db consumer {}",this.name);
}finally{
databaseConnection=null;
}
throw sqlException;
}catch(Exception exception){
logger.error("An error ocurred while processing consumed item.", exception);
throw exception;
}
The prepared statement is a local variable, not a function one, so it can be re-used with every loop (this piece is part of a function that is called once for every result in a query that is done with a prepared statement to the same db connection). The error occurs when I'm trying to work on the prepared statement.
And the latest stack:
18/06/15 15:54:34.841 [ecbcbcmt] TRACE DatabaseConsumer - creating statement 18/06/15 15:54:34.850 [ecbcbcmt] ERROR DatabaseConsumer - An
error ocurred while processing consumed item. java.sql.SQLException:
You can't operate on a closed Statement!!! at
com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
~[mchange-commons-java-0.2.9.jar:0.2.9] at
com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:77)
~[mchange-commons-java-0.2.9.jar:0.2.9] at
com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.setString(NewProxyPreparedStatement.java:488)
~[c3p0-0.9.5.jar:0.9.5] at
us.inswitch.jticket.subscriber.consumer.database.DatabaseConsumer.processRow(DatabaseConsumer.java:152)
[bin/:na] at
us.inswitch.jticket.subscriber.consumer.database.DatabaseNumberConsumer.processRows(DatabaseNumberConsumer.java:73)
[bin/:na] at
us.inswitch.jticket.subscriber.consumer.database.DatabaseConsumer.start(DatabaseConsumer.java:65)
[bin/:na] at
us.inswitch.jticket.subscriber.consumer.Consumer.run(Consumer.java:35)
[bin/:na] at java.lang.Thread.run(Thread.java:662) [na:1.6.0_38-ea]
Caused by: java.lang.NullPointerException: null at
com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.setString(NewProxyPreparedStatement.java:482)
~[c3p0-0.9.5.jar:0.9.5] ... 5 common frames omitted 18/06/15
15:54:34.852 [ecbcbcmt] WARN DatabaseConsumer - An error ocurred
while consuming table data. DatabaseConsumer will be restarted.
java.sql.SQLException: You can't operate on a closed Statement!!! at
com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
~[mchange-commons-java-0.2.9.jar:0.2.9] at
com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:77)
~[mchange-commons-java-0.2.9.jar:0.2.9] at
com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.setString(NewProxyPreparedStatement.java:488)
~[c3p0-0.9.5.jar:0.9.5] at
us.inswitch.jticket.subscriber.consumer.database.DatabaseConsumer.processRow(DatabaseConsumer.java:152)
~[bin/:na] at
us.inswitch.jticket.subscriber.consumer.database.DatabaseNumberConsumer.processRows(DatabaseNumberConsumer.java:73)
~[bin/:na] at
us.inswitch.jticket.subscriber.consumer.database.DatabaseConsumer.start(DatabaseConsumer.java:65)
~[bin/:na] at
us.inswitch.jticket.subscriber.consumer.Consumer.run(Consumer.java:35)
[bin/:na] at java.lang.Thread.run(Thread.java:662) [na:1.6.0_38-ea]
Caused by: java.lang.NullPointerException: null at
com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.setString(NewProxyPreparedStatement.java:482)
~[c3p0-0.9.5.jar:0.9.5] ... 5 common frames omitted
It's weird because it's completely random. We've had the program up and running for a while now, with all the connections working perfectly fine, and we get this kind of problem.

Use local variables, or otherwise use PreparedStatements carefully.
This avoids concurrent usage as in a web application.
try (PreparedStatement stmt = conn.prepareStatement(sql)) {
...
try (ResultSet rs = stmt.executeQuery()) {
...
} // rs close
} // Does stm.close()
The try-with-resources also closes when an exception is thrown.

The statement stmtLasProcessedTransaction.close(); is present at two places which is fishy!!
1. within try block in conditional if
2. within catch block
Better programming practice is to have only one close statement, that too in finally block
PS: You can have try-catch-finally within finally block as well

JPA: PersistenceException, could not update

I'm using Play Framework 1.2.5 with Java but this is more of a JPA question.
In my project I often search users by their email, so I created the following method:
public static User getUserByEmail(String email) {
User user = User.find("email = ?", email).first();
return user;
}
and I call it like this from various methods:
User user = User.getUserByEmail("test#email.com");
When I'm trying to modify a field of User like this:
User user = User.getUserByEmail("kospol#test.com");
user.name = "kospol";
user.save();
I often get a the following exception resulting in total freeze:
Execution exception (In /app/controllers/*******.java around line 46)
PersistenceException occured : update User set activated=?, ... registered=?, registeredFrom=?, version=? where id=? and version=?
play.exceptions.JavaExecutionException: update User set activated=?, ... registered=?, registeredFrom=?, version=? where id=? and version=?
at play.mvc.ActionInvoker.invoke(ActionInvoker.java:237)
at Invocation.HTTP Request(Play!)
Caused by: javax.persistence.PersistenceException: update User set activated=?, ... registered=?, registeredFrom=?, version=? where id=? and version=?
at play.db.jpa.JPABase._save(JPABase.java:44)
at play.db.jpa.GenericModel.save(GenericModel.java:204)
at controllers.PushService.register(PushService.java:46)
at play.mvc.ActionInvoker.invokeWithContinuation(ActionInvoker.java:557)
at play.mvc.ActionInvoker.invoke(ActionInvoker.java:508)
at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:484)
at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:479)
at play.mvc.ActionInvoker.invoke(ActionInvoker.java:161)
... 1 more
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not update: [models.User#3606]
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1389)
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1317)
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1323)
at org.hibernate.ejb.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:965)
at play.db.jpa.JPABase._save(JPABase.java:41)
... 8 more
Caused by: org.hibernate.exception.GenericJDBCException: could not update: [models.User#3606]
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:140)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:128)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2613)
at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2495)
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2822)
at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:113)
at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:273)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:265)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:185)
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:345)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216)
at org.hibernate.ejb.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:962)
... 9 more
Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4096)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4028)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2490)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2651)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2683)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2144)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2444)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2362)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2347)
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2595)
... 19 more
I have added the #Version annotation to the model. It seems that the problem is the Lock wait timeout exceeded. How can I avoid this?
I have tried to use the .merge() method with no results and I'm thinking to remove the getUserByEmail method and get the model straight.

Disclaimer : it's not easy to answer your question without having the whole project, but here are 2 potential issues to investigate.
I'm pretty sure that the problem isn't caused by the method getUserByEmail itself. (especially because sometimes it works and sometime not).
The problem is caused by another very heavy query locking the User table (or maybe only the row your searching for) for a very long time (exceeding the wait timeout).
So investigate the heavy queries (one way to do that is to log all sql queries and see queries that are running when you got the exception).
Another option is a deadlock (if you don't see any long queries) : here is a typical scenario.
Thread 1 needs to lock table User and table B. thread 2 needs to lock table B and table User.
At time 0:
thread 1 receive a lock on table User
thread 2 receive a lock on table B
At time 1:
thread 1 block and wait for a lock on table B
thread 2 block and wait for a lock on table User
At time 2:
wait lock timeout occurs for one thread, the lock is released and the exception is thrown.
At time 3:
the other thread receive finally the required lock and execute it's query.

Did you make a try with explicit save?
According with Play framework persistence docs you can do something like this:
public static void save(Long id) {
User user = User.findById(id);
user.edit(params);
validation.valid(user);
if(validation.hasErrors()) {
// Here we have to explicitly discard the user modifications...
user.refresh();
edit(id);
}
show(id);
}
Maybe you have to use edit() method (I am very new to play framework and do not know if works).
Hope this helps.

If i am in the right track this is the problem you are facing..!!
The problem is With the getUserByEmail function that you have added inside the JPA.
Entity Class is always checked in full, so if you want to create a
some variables or functions inside the JPA. You must tell what the
Program must do with the function or variable. For that you can use
annotations
Here in your case you must tell the program to ignore the function. For that you can use
#Transient
This annotation specifies that the property or field is not persistent. It is used to annotate a property or field of an entity class, mapped superclass, or embeddable class.
In you case you can use this in your JPA
#Transient
public static User getUserByEmail(String email) {
User user = User.find("email = ?", email).first();
return user;
}

java thread hangs while using executorService executing a oracle query

"pool-11-thread-1" prio=10 tid=0x0a974c00 nid=0x7210 runnable [0x3f3ad000]
java.lang.Thread.State: RUNNABLE
at oracle.jdbc.driver.T2CStatement.t2cDefineExecuteFetch(Native Method)
at oracle.jdbc.driver.T2CPreparedStatement.doDefineExecuteFetch(T2CPreparedStatement.java:878)
at oracle.jdbc.driver.T2CPreparedStatement.executeForRows(T2CPreparedStatement.java:760)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1062)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1126)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3339)
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3445)
- locked <0x69579fb0> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x66157d68> (a oracle.jdbc.driver.T2CConnection)
at org.jboss.resource.adapter.jdbc.CachedPreparedStatement.execute(CachedPreparedStatement.java:216)
at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:209)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:180)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery(GeneralStatement.java:205)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralStatement.java:173)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryForObject(GeneralStatement.java:104)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:561)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:536)
at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForObject(SqlMapSessionImpl.java:93)
at org.springframework.orm.ibatis.SqlMapClientTemplate$1.doInSqlMapClient(SqlMapClientTemplate.java:273)
at org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:209)
at org.springframework.orm.ibatis.SqlMapClientTemplate.queryForObject(SqlMapClientTemplate.java:271)
at com.alipay.bipgw.common.dal.bankchannel.ibatis.IbatisBipBusiOrderDAO.queryOrderOutTime(IbatisBipBusiOrderDAO.java:319)
at sun.reflect.GeneratedMethodAccessor3333.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at com.alipay.bipgw.common.dal.monitor.DalMonitorInterceptor.invoke(DalMonitorInterceptor.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy79.queryOrderOutTime(Unknown Source)
at com.alipay.bipgw.prodcore.repository.impl.BusiOrderRepositoryImpl.queryOrderOutTime(BusiOrderRepositoryImpl.java:402)
at com.alipay.bipgw.prodcore.listener.ProdStatusChangeTimeoutTaskListener.execute(ProdStatusChangeTimeoutTaskListener.java:148)
at com.alipay.bipgw.prodcore.listener.ProdStatusChangeTimeoutTaskListener.access$000(ProdStatusChangeTimeoutTaskListener.java:60)
at com.alipay.bipgw.prodcore.listener.ProdStatusChangeTimeoutTaskListener$1.run(ProdStatusChangeTimeoutTaskListener.java:104)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
the java programm like this :
public void onUniformEvent(UniformEvent message, UniformEventContext uContext) {
try {
// single thread running
service.execute(new Runnable() {
public void run() {
try{
execute();
}catch(Exception e ){
logger.error("working error",e);
}
}
});
} catch (RejectedExecutionException e) {
logger.error("ProdStatusChangeTimeoutTaskListener:error", e);
} catch (Exception e) {
logger.error("ProdStatusChangeTimeoutTaskListener:error", e);
}
}
//omit the body
private void execute() {.....}
and the execute method will not start any thread.
in two days i dump several thread dump
2013-03-04 16:54:12
- locked <0x695f91f0> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x6615a2d0> (a oracle.jdbc.driver.T2CConnection)
2013-03-04 17:20:53
- locked <0x695f91f0> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x6615a2d0> (a oracle.jdbc.driver.T2CConnection)
2013-03-05 10:58:30
- locked <0x6957bec8> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x66157e90> (a oracle.jdbc.driver.T2CConnection)
2013-03-05 17:16:31
- locked <0x69579fb0> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x66157d68> (a oracle.jdbc.driver.T2CConnection)
seems like the lock hold by jdbc client has changed, but the first two in 2013-03-04 16:54:12 and 2013-03-04 17:20:53, they are the same
I am using a Excutors.newSingleThreadExecutor() doing a query job in backgroud, and the following task will be submit to this executor Service in 20 minutes interval, but the work thread seems to hangs while executing the query , so the following task will not be executed. It last for several days , no exception occur and no log output at all, somebody can help me ? thanks

The problem is likely to lie in your actual query - is it a long running task? Do you know how long on average it takes to complete?
Its possible (but this is dependent on the query itself) that a previous query is locking tables which in turn block later ones.
The first thing I would do it to verify that the query does indeed complete within 20 minutes, and does so consistently. After you know that it does you can then investigate other possible causes for the hanging behavior you're seeing.
When the query is running I'd also suggest that you check the explain plan for your query (see what its doing - whats taking the most time etc). I'd also check that the relevant statistics are up to date (perhaps less of an issue on newer versions of Oracle)

This problem has been solved~
after several real time executes , out dba monitor the db server side, and find that the query time is 210s which execeeds the query-out-time(180s), but the cancel request of the stamentent may not be effective and the db server may not response to that request . so the thread which porform the original query hangs. the similar situation see [When I call PreparedStatement.cancel() in a JDBC application, does it actually kill it in an Oracle database?
the solution can been done by following:
add sqlprofile to the target table，and make sql using skip scan to short the query time.
but the deepest reason why hanging will occur is still unknown.