Develop Reference

Deadlock while using ORMLite

I have a multithreaded server Java application which receives requests and does queries/updates to a Postgres DB through OrmLite. Under load, several requests come in which are interested in the same DB row. Thread1 might select, change values and then update. At the same time Thread2 tries something similar. This is currently not synchronized and not done inside a transaction. Without surprise, the update of Thread1, might not be seen by Thread2. That's OK (Thread2 can overwrite results from Thread1) and is not my problem.
However, when running this application for some time, I get to a deadlock situation, which results in all available DB connections being used up (and then crash). It seems it is not a standard deadlock (with a circular lock dependency), instead most threads are waiting on a lock, and the thread holding this lock seems to be waiting for a socket read (which probably does not happen, see below).
Using
OrmLite 5.1,
JVM is Java 1.8.0_251 Hotspot Client VM,
Postgres JDBC 42.2.9
How should I go forward to fix this?
Below are relevant parts of the thread dump (analyzed by https://spotify.github.io/threaddump-analyzer )
The thread holding the main lock (0x00000000c0179e18), seems to be waiting on a socket:
"RaspService-2089": running, holding [0x00000000c0179e18, 0x00000000c2c1f6c0]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:140)
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:109)
at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:67)
at org.postgresql.core.PGStream.receiveChar(PGStream.java:335)
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:505)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:141)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:211)
at org.postgresql.Driver.makeConnection(Driver.java:458)
at org.postgresql.Driver.connect(Driver.java:260)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at com.j256.ormlite.jdbc.JdbcConnectionSource.makeConnection(JdbcConnectionSource.java:266)
at com.j256.ormlite.jdbc.JdbcPooledConnectionSource.getReadWriteConnection(JdbcPooledConnectionSource.java:140)
at com.j256.ormlite.dao.BaseDaoImpl.update(BaseDaoImpl.java:408)
at vgs.vigi.servlet.OrmLite.update(OrmLite.java:361)
at vgs.vigi.servlet.CachedDao.update(CachedDao.java:287)
at vgs.vigi.ble.RaspClient.run(RaspClient.java:177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
26 threads waiting to free connections wait on that lock with stacks like:
"pool-4-thread-96": waiting to acquire [0x00000000c0179e18], holding [0x00000000c0b250a8]
at com.j256.ormlite.jdbc.JdbcPooledConnectionSource.releaseConnection(JdbcPooledConnectionSource.java:168)
at com.j256.ormlite.dao.BaseDaoImpl.create(BaseDaoImpl.java:331)
at vgs.vigi.servlet.OrmLite.create(OrmLite.java:181)
at vgs.vigi.servlet.CachedDao.create(CachedDao.java:126)
at vgs.vigi.logic.Notification.sendNotification(Notification.java:491)
at vgs.vigi.logic.Notification$1.run(Notification.java:640)
at vgs.lib.MyTimer$2.run(MyTimer.java:103)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
More threads waiting to release connections
"RaspService-828": waiting to acquire [0x00000000c0179e18], holding [0x00000000c1187f88]
at com.j256.ormlite.jdbc.JdbcPooledConnectionSource.releaseConnection(JdbcPooledConnectionSource.java:168)
at com.j256.ormlite.dao.BaseDaoImpl.update(BaseDaoImpl.java:412)
at vgs.vigi.servlet.OrmLite.update(OrmLite.java:361)
at vgs.vigi.servlet.CachedDao.update(CachedDao.java:287)
at vgs.vigi.ble.CmdRaspExcutor$8.exec(CmdRaspExcutor.java:318)
at vgs.vigi.ble.RaspClient.run(RaspClient.java:182)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
While many also try to acquire connections:
"RaspService-991": waiting to acquire [0x00000000c0179e18], holding [0x00000000c1624058]
at com.j256.ormlite.jdbc.JdbcPooledConnectionSource.getReadWriteConnection(JdbcPooledConnectionSource.java:125)
at com.j256.ormlite.dao.BaseDaoImpl.update(BaseDaoImpl.java:408)
at vgs.vigi.servlet.OrmLite.update(OrmLite.java:361)
at vgs.vigi.servlet.CachedDao.update(CachedDao.java:287)
at vgs.vigi.ble.RaspClient.run(RaspClient.java:177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Some GC is happening (inconsistent means Thread is "BLOCKED (on object monitor)" without waiting for anything)
"qtp1719311117-931": inconsistent?, holding [0x00000000eabfc510]
at java.lang.Runtime.gc(Native Method)
at java.lang.System.gc(System.java:993)
at vgs.vigi.servlet.OrmLite.clearCache(OrmLite.java:33)
at vgs.vigi.servlet.OrmLite.dao(OrmLite.java:215)
at vgs.vigi.servlet.OrmLite.getAll(OrmLite.java:300)
at vgs.vigi.servlet.CachedDao.getAll(CachedDao.java:227)
at vgs.lib.Ajax.sGetAll(Ajax.java:101)
...
And also GC in another thread (which is explicitely coded in our code - not sure why though)
"RaspService-1882": running, holding [0x00000000c05c84c8, 0x00000000c2bed7b0]
at java.lang.Runtime.gc(Native Method)
at java.lang.System.gc(System.java:993)
at vgs.vigi.servlet.OrmLite.clearCache(OrmLite.java:33)
at vgs.vigi.servlet.OrmLite.dao(OrmLite.java:215)
at vgs.vigi.servlet.OrmLite.update(OrmLite.java:360)
at vgs.vigi.servlet.CachedDao.update(CachedDao.java:285)
at vgs.vigi.ble.RaspClient.run(RaspClient.java:177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Can I expect OrmLite to be safe with a multithreaded approach as described?
Are there best practices to avoid this issue (while still keeping the multithreaded nature of the server)?
Update
I have a thread dump of a second run, which looks a bit different.
Here the thread holding the lock that everyone is waiting for is inconsistent
"RaspService-1405": inconsistent?, holding [0x00000000c01da9b8, 0x00000000c1cfed28]
With a raw stack of:
"RaspService-1405" #1469 prio=5 os_prio=0 tid=0x0000000021579800 nid=0xa2f4 waiting for monitor entry [0x000000002b36e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.j256.ormlite.jdbc.JdbcPooledConnectionSource.getReadWriteConnection(JdbcPooledConnectionSource.java:125)
- locked <0x00000000c01da9b8> (a java.lang.Object)
at com.j256.ormlite.dao.BaseDaoImpl.update(BaseDaoImpl.java:408)
at vgs.vigi.servlet.OrmLite.update(OrmLite.java:361)
at vgs.vigi.servlet.CachedDao.update(CachedDao.java:287)
at vgs.vigi.ble.RaspClient.run(RaspClient.java:177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x00000000c1cfed28> (a java.util.concurrent.ThreadPoolExecutor$Worker)
There is also a RUNNING thread which reads from a connection. Not sure whether that is blocked:
"RaspService-1410": running, holding [0x00000000ed1877c8, 0x00000000c1cfe208]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:140)
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:109)
at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:67)
at org.postgresql.core.PGStream.receiveChar(PGStream.java:335)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2008)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:310)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:447)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:368)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:158)
at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:124)
at com.j256.ormlite.jdbc.JdbcDatabaseConnection.update(JdbcDatabaseConnection.java:294)
at com.j256.ormlite.jdbc.JdbcDatabaseConnection.update(JdbcDatabaseConnection.java:217)
at com.j256.ormlite.stmt.mapped.MappedUpdate.update(MappedUpdate.java:101)
at com.j256.ormlite.stmt.StatementExecutor.update(StatementExecutor.java:472)
at com.j256.ormlite.dao.BaseDaoImpl.update(BaseDaoImpl.java:410)
at vgs.vigi.servlet.OrmLite.update(OrmLite.java:361)
at vgs.vigi.servlet.CachedDao.update(CachedDao.java:287)
at vgs.vigi.ble.RaspClient.run(RaspClient.java:177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Update 2:
Here's a graph of sessions as seen from pgAdmin

seems to be waiting on a socket:
It waits until new connection is created, also holding lock inside pool.
Do you have session limit in Postgres? If you do I suggest you set it slightly bigger than pool size in Java.
Otherwise, it is easy to have deadlock if pool size is equal to sessions limit size
All connections are taken (Java pool limit is reached, session count limit is reached)
Application tries to get new connection, takes pool lock and blocked by PG
Application tries to release connection, it cannot take pool lock so cannot release connection to PG, so session limit is still reached

Blocked threads due to long running connection?

Every now and then an old Java I have to maintain stops responding. I managed to get a couple of thread stack traces and most threads are blocked like this trying to obtain a connection:
"tomcat-http-8180-168" - Thread t#10137
java.lang.Thread.State: BLOCKED
at oracle.jdbc.pool.OracleImplicitConnectionCache.retrieveCacheConnection(OracleImplicitConnectionCache.java:566)
- waiting to lock <566080> (a oracle.jdbc.pool.OracleImplicitConnectionCache) owned by "Thread-6" t#29
The thread holding the lock shows this:
"Thread-6" - Thread t#29
java.lang.Thread.State: BLOCKED
at oracle.jdbc.driver.PhysicalConnection.closeLogicalConnection(PhysicalConnection.java:3849)
- waiting to lock <146da30> (a oracle.jdbc.driver.T4CConnection) owned by "tomcat-http-8180-369" t#17665
at oracle.jdbc.driver.LogicalConnection.cleanupAndClose(LogicalConnection.java:304)
at oracle.jdbc.pool.OracleImplicitConnectionCache.closeCheckedOutConnection(OracleImplicitConnectionCache.java:1392)
at oracle.jdbc.pool.OracleImplicitConnectionCacheThread.runAbandonedTimeout(OracleImplicitConnectionCacheThread.java:250)
- locked <566080> (a oracle.jdbc.pool.OracleImplicitConnectionCache)
at oracle.jdbc.pool.OracleImplicitConnectionCacheThread.run(OracleImplicitConnectionCacheThread.java:81)
This thread seems to be responsible for closing abandoned connections, and it's itself blocked waiting for this other thread to finish:
"tomcat-http-8180-369" - Thread t#17665
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Packet.java:282)
at oracle.net.ns.DataPacket.receive(DataPacket.java:103)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:230)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:175)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:100)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:85)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:122)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:78)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readB1(T4CSocketInputStreamWrapper.java:149)
at oracle.jdbc.driver.T4CMAREngine.buffer2Value(T4CMAREngine.java:2393)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB8(T4CMAREngine.java:1401)
at oracle.jdbc.driver.T4C8TTILob.readRPA(T4C8TTILob.java:837)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:292)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)
at oracle.jdbc.driver.T4C8TTIClob.read(T4C8TTIClob.java:240)
at oracle.jdbc.driver.T4CConnection.getChars(T4CConnection.java:3015)
- locked <146da30> (a oracle.jdbc.driver.T4CConnection)
at oracle.sql.CLOB.getChars(CLOB.java:402)
at oracle.jdbc.driver.OracleClobReader.needChars(OracleClobReader.java:187)
at oracle.jdbc.driver.OracleClobReader.read(OracleClobReader.java:142)
at java.io.Reader.read(Reader.java:123)
at oracle.jdbc.driver.ClobAccessor.getString(ClobAccessor.java:291)
at oracle.jdbc.driver.T4CClobAccessor.getString(T4CClobAccessor.java:481)
at oracle.jdbc.driver.OracleResultSetImpl.getString(OracleResultSetImpl.java:1251)
- locked <146da30> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OracleResultSet.getString(OracleResultSet.java:494)
This last thread is running a heavy query. Which is also why it's considered abandoned and Tomcat is trying to close it but it seemingly can't as it's in use and has a lock.
I don't understand why:
Oracle can't close the connection.
The other threads can't get a connection from the pool until the abandoned connection is closed.
Because by looking at the thread stack above it's what's happening. When the extra-long query finished (I know I have to look into that query) then the app started responding again as the threads unblocked.
This is Tomcat's (v6) pool config (sensitive details ommitted):
<Resource
name="jdbc/MainDBPool"
AutoCommit="true"
defaultReadOnly="false"
driverClassName="oracle.jdbc.OracleDriver"
factory="oracle.jdbc.pool.OracleDataSourceFactory"
fairQueue="false"
initialSize="10"
jdbcInterceptors="ConnectionState;StatementFinalizer"
jmxEnabled="true"
logAbandoned="false"
maxActive="100"
maxIdle="75"
maxWait="30000"
minEvictableIdleTimeMillis="5000"
minIdle="10"
removeAbandoned="true"
removeAbandonedTimeout="60"
testOnBorrow="false"
testOnReturn="false"
testWhileIdle="false"
timeBetweenEvictionRunsMillis="5000"
type="oracle.jdbc.pool.OracleDataSource"
connectionCachingEnabled="true"
connectionCacheName="tomcatConnectionCache1"
fastConnectionFailoverEnabled="true"
implicitCachingEnabled="true"
connectionCacheProperties="(ValidateConnection=true, PropertyCheckInterval=60, AbandonedConnectionTimeout=300, InitialLimit=10, MinLimit=30, MaxLimit=200, ConnectionWaitTimeout=30, InactivityTimeout=300)"
useEquals="false"
validationInterval="30000"
/>
I also looked into other possible causes, like a long full GC, but GC logging was enabled and there isn't a long pause that would explain this.
Thanks in advance.

Oracle Implicit Connection Cache is desupported. You must use Universal Connection Pool (UCP) which is the ICC's replacement. You can download ucp.jar and check out "UCP with Tomcat" whitepaper for more details.

java thread hangs while using executorService executing a oracle query

"pool-11-thread-1" prio=10 tid=0x0a974c00 nid=0x7210 runnable [0x3f3ad000]
java.lang.Thread.State: RUNNABLE
at oracle.jdbc.driver.T2CStatement.t2cDefineExecuteFetch(Native Method)
at oracle.jdbc.driver.T2CPreparedStatement.doDefineExecuteFetch(T2CPreparedStatement.java:878)
at oracle.jdbc.driver.T2CPreparedStatement.executeForRows(T2CPreparedStatement.java:760)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1062)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1126)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3339)
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3445)
- locked <0x69579fb0> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x66157d68> (a oracle.jdbc.driver.T2CConnection)
at org.jboss.resource.adapter.jdbc.CachedPreparedStatement.execute(CachedPreparedStatement.java:216)
at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:209)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:180)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery(GeneralStatement.java:205)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralStatement.java:173)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryForObject(GeneralStatement.java:104)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:561)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:536)
at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForObject(SqlMapSessionImpl.java:93)
at org.springframework.orm.ibatis.SqlMapClientTemplate$1.doInSqlMapClient(SqlMapClientTemplate.java:273)
at org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:209)
at org.springframework.orm.ibatis.SqlMapClientTemplate.queryForObject(SqlMapClientTemplate.java:271)
at com.alipay.bipgw.common.dal.bankchannel.ibatis.IbatisBipBusiOrderDAO.queryOrderOutTime(IbatisBipBusiOrderDAO.java:319)
at sun.reflect.GeneratedMethodAccessor3333.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at com.alipay.bipgw.common.dal.monitor.DalMonitorInterceptor.invoke(DalMonitorInterceptor.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy79.queryOrderOutTime(Unknown Source)
at com.alipay.bipgw.prodcore.repository.impl.BusiOrderRepositoryImpl.queryOrderOutTime(BusiOrderRepositoryImpl.java:402)
at com.alipay.bipgw.prodcore.listener.ProdStatusChangeTimeoutTaskListener.execute(ProdStatusChangeTimeoutTaskListener.java:148)
at com.alipay.bipgw.prodcore.listener.ProdStatusChangeTimeoutTaskListener.access$000(ProdStatusChangeTimeoutTaskListener.java:60)
at com.alipay.bipgw.prodcore.listener.ProdStatusChangeTimeoutTaskListener$1.run(ProdStatusChangeTimeoutTaskListener.java:104)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
the java programm like this :
public void onUniformEvent(UniformEvent message, UniformEventContext uContext) {
try {
// single thread running
service.execute(new Runnable() {
public void run() {
try{
execute();
}catch(Exception e ){
logger.error("working error",e);
}
}
});
} catch (RejectedExecutionException e) {
logger.error("ProdStatusChangeTimeoutTaskListener:error", e);
} catch (Exception e) {
logger.error("ProdStatusChangeTimeoutTaskListener:error", e);
}
}
//omit the body
private void execute() {.....}
and the execute method will not start any thread.
in two days i dump several thread dump
2013-03-04 16:54:12
- locked <0x695f91f0> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x6615a2d0> (a oracle.jdbc.driver.T2CConnection)
2013-03-04 17:20:53
- locked <0x695f91f0> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x6615a2d0> (a oracle.jdbc.driver.T2CConnection)
2013-03-05 10:58:30
- locked <0x6957bec8> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x66157e90> (a oracle.jdbc.driver.T2CConnection)
2013-03-05 17:16:31
- locked <0x69579fb0> (a oracle.jdbc.driver.T2CPreparedStatement)
- locked <0x66157d68> (a oracle.jdbc.driver.T2CConnection)
seems like the lock hold by jdbc client has changed, but the first two in 2013-03-04 16:54:12 and 2013-03-04 17:20:53, they are the same
I am using a Excutors.newSingleThreadExecutor() doing a query job in backgroud, and the following task will be submit to this executor Service in 20 minutes interval, but the work thread seems to hangs while executing the query , so the following task will not be executed. It last for several days , no exception occur and no log output at all, somebody can help me ? thanks

The problem is likely to lie in your actual query - is it a long running task? Do you know how long on average it takes to complete?
Its possible (but this is dependent on the query itself) that a previous query is locking tables which in turn block later ones.
The first thing I would do it to verify that the query does indeed complete within 20 minutes, and does so consistently. After you know that it does you can then investigate other possible causes for the hanging behavior you're seeing.
When the query is running I'd also suggest that you check the explain plan for your query (see what its doing - whats taking the most time etc). I'd also check that the relevant statistics are up to date (perhaps less of an issue on newer versions of Oracle)

This problem has been solved~
after several real time executes , out dba monitor the db server side, and find that the query time is 210s which execeeds the query-out-time(180s), but the cancel request of the stamentent may not be effective and the db server may not response to that request . so the thread which porform the original query hangs. the similar situation see [When I call PreparedStatement.cancel() in a JDBC application, does it actually kill it in an Oracle database?
the solution can been done by following:
add sqlprofile to the target table，and make sql using skip scan to short the query time.
but the deepest reason why hanging will occur is still unknown.

c3p0 hangs in awaitAvailable with hibernate

I have console application that hangs during execution. Here is my configuration:
cfg.setProperty("hibernate.connection.driver_class", "com.mysql.jdbc.Driver");
cfg.setProperty("hibernate.connection.url", "jdbc:mysql://localhost:3306/db?user=db&password=db");
cfg.setProperty("hibernate.connection.username", "db");
cfg.setProperty("hibernate.connection.password", "db");
cfg.setProperty("hibernate.connection.pool_size", "5");
cfg.setProperty("hibernate.connection.autocommit", "false");
cfg.setProperty("hibernate.c3p0.min_size", "5");
cfg.setProperty("hibernate.c3p0.max_size", "20");
cfg.setProperty("hibernate.c3p0.timeout", "300");
cfg.setProperty("hibernate.c3p0.max_statements", "50");
cfg.setProperty("hibernate.c3p0.idle_test_period", "3000");
Here is my stacktrace:
"main" prio=10 tid=0x000000000168f800 nid=0x1c37 in Object.wait() [0x00007fa60d0ad000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007400f4c68> (a com.mchange.v2.resourcepool.BasicResourcePool)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
- locked <0x00000007400f4c68> (a com.mchange.v2.resourcepool.BasicResourcePool)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:84)
at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:281)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:297)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169)
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:160)
at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1392)
at org.kriyak.parser.IndexArchiveRapid.indexFile(IndexArchiveRapid.java:70)
at org.kriyak.parser.IndexArchiveRapid.main(IndexArchiveRapid.java:53)
I open only one conencton and it doesn't seem that I leak them. And also I use one thread. I haven't adjusted any mysql settings except memory usage. Mysql works fine from console. Why can this happen? Is this c3p0 error?

does this happen immediately, or after a while? that is, do checkouts initially succeed, but then hang like this? if so, it looks like a Connection leak. please try setting c3p0 params unreturnedConnectionTimeout and debugUnreturnedConnectionStackTraces to see if there is a leak. See
http://www.mchange.com/projects/c3p0/#configuring_to_debug_and_workaround_broken_clients , http://www.mchange.com/projects/c3p0/#unreturnedConnectionTimeout , http://www.mchange.com/projects/c3p0/#debugUnreturnedConnectionStackTraces .
if this happens immediately, if no Connections are successfully checked out, the question is whether the pool ever succeeds at acquiring Connections. by default, if it never does succeed, after about 30 seconds your thread should break with a failure. (it doesn't look like you've done this, but if for example you'd set acquireRetryAttempts to zero, c3p0 might hang indefinitely waiting for Connections.)
to debug c3p0 issues, it's helpful to capture the version and config information that c3p0 dumps to logs at INFO level on pool initialization.
good luck!

Also, you haven't seem to initialize the checkoutTime parameter for c3p0, which specifies the amount of time a client should wait for acquiring a connection from the connection pool.
see http://www.mchange.com/projects/c3p0/#checkoutTimeout

You may need to increase c3p0.numHelperThreads. The helper threads are responsible for creating new database connections and adding them to the pool. If there aren't enough threads to keep up with demand then application threads will be stuck waiting in awaitAvailable(). To confirm this is the case look at all the HelperThreads and see if they are all in use and in the process of connecting to the database.

how do I solve deadlock of proxy example trafficlock and ssl handshakelock?

I combined large parts of the netty examples HexdumpProxy (http://netty.io/docs/stable/xref/org/jboss/netty/example/proxy/) and SecureChat (http://netty.io/docs/stable/xref/org/jboss/netty/example/securechat) to form an SSL (and non-SSL) capable proxy (to a non-SSL back end). This seemed like a good idea, proved to be simple enough and it is exactly what I need, currently.
The proxy example code uses a trafficlock lock as a solution to a race condition reported in 2010 (http://markmail.org/message/x7jc6mqx6ripynqf) happening around saturated channels and setting the writeable and readable state of the incoming and outgoing channels.
Now, in my combined example, under higher load, this leads to a deadlock because another lock "handshakelock" in the SSL code is intertwined. See the profiler diagnostic output below.
I fear that, even after reading the original discussion, I do not understand the underlying issue with the traffic lock good enough to find a straightforward solution for this deadlock.
(this was with netty 3.2.6)
Java-level deadlock has been detected
This means that some threads are blocked waiting to enter a synchronization block or
waiting to reenter a synchronization block after an Object.wait() call, where each thread
owns one monitor while trying to obtain another monitor already held by another thread.
Deadlock:
New I/O client worker #1-2 is waiting to lock java.lang.Object#7900f3c9 which is held by New I/O server worker #1-2
New I/O server worker #1-2 is waiting to lock java.lang.Object#2d854f2f which is held by New I/O client worker #1-2
Thread stacks
New I/O client worker #1-2 [BLOCKED; waiting to lock java.lang.Object#7900f3c9]
org.jboss.netty.handler.ssl.SslHandler.wrap(SslHandler.java:665) <== sync handshakelock
org.jboss.netty.handler.ssl.SslHandler.handleDownstream(SslHandler.java:461)
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:776)
org.jboss.netty.channel.Channels.write(Channels.java:632)
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
org.jboss.netty.channel.Channels.write(Channels.java:611)
org.jboss.netty.channel.Channels.write(Channels.java:578)
org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
com.activevideo.frontend.ProxyInboundHandler$OutboundHandler.messageReceived(ProxyInboundHandler.java:162) <== sync trafficlock
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(unknown source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(unknown source)
java.lang.Thread.run(unknown source)
New I/O server worker #1-2 [BLOCKED; waiting to lock java.lang.Object#2d854f2f]
com.activevideo.frontend.ProxyInboundHandler.channelInterestChanged(ProxyInboundHandler.java:138) <== sync trafficlock
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:116)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelInterestChanged(SimpleChannelUpstreamHandler.java:192)
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:116)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelInterestChanged(SimpleChannelUpstreamHandler.java:192)
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:116)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelInterestChanged(SimpleChannelUpstreamHandler.java:192)
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:116)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
org.jboss.netty.channel.Channels.fireChannelInterestChanged(Channels.java:335)
org.jboss.netty.channel.socket.nio.NioWorker.setInterestOps(NioWorker.java:728)
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:129)
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
org.jboss.netty.handler.ssl.SslHandler.handleDownstream(SslHandler.java:430)
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:776)
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:60)
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
org.jboss.netty.channel.Channels.setInterestOps(Channels.java:652)
org.jboss.netty.channel.AbstractChannel.setInterestOps(AbstractChannel.java:222)
org.jboss.netty.channel.AbstractChannel.setReadable(AbstractChannel.java:244)
com.activevideo.frontend.SSLProxyInboundHandler$BackendConnector$1.operationComplete(SSLProxyInboundHandler.java:92)
org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:381)
org.jboss.netty.channel.DefaultChannelFuture.addListener(DefaultChannelFuture.java:148)
com.activevideo.frontend.SSLProxyInboundHandler$BackendConnector.operationComplete(SSLProxyInboundHandler.java:87)
org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:381)
org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:367)
org.jboss.netty.channel.DefaultChannelFuture.setSuccess(DefaultChannelFuture.java:316)
org.jboss.netty.handler.ssl.SslHandler.setHandshakeSuccess(SslHandler.java:1040)
org.jboss.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:838)
org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:907) <=== sync handshakelock
org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:620)
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214)
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(unknown source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(unknown source)
java.lang.Thread.run(unknown source)

Okay, I think I fixed it.
The solution to the earlier race condition in the Proxy example unnecessarily includes the call to write() in the synchronized block.
The original problem there was this (rare) race condition between the outgoing thread (TO) and incoming thread (TI):
TO: inboundChannel.write() (at messageReceived)
TO: inboundChannel.isWritable() returns false (at messageReceived)
Then pending writes issued at (1) are flushed
TI: inboundChannel.isWritable() returns true (at channelInterestChanged)
TI: outboundChannel.setReadable(true) (at channelInterestChanged)
TO: outboundChannel.setReadable(false) (at messageReceived)
The solution in 2010 was to introduce synchronizing (using 'trafficlock') around setting the readable flag in the messageReceived() handler of both the inboundChannel and outboundChannel (and also in the InterestChanged handler), like so:
synchronized (trafficLock) {
outboundChannel.write(msg);
// If outboundChannel is saturated, do not read until notified in
// OutboundHandler.channelInterestChanged().
if (!outboundChannel.isWritable()) {
e.getChannel().setReadable(false);
}
}
This indeed solves the race condition, as you prevent steps 3, 4 and 5 to interfere with the steps 2 and 6. However, it is safe to leave step 1, the write(), out of the synchronized block.
The call to write() caused the deadlock, because way down that call in the SSLHandler, it uses another lock handshakelock.
So.. I moved the call to write() outside the synchronized block in both locations. The deadlock is now gone. I suggest the 'official' proxy example is changed accordingly.