Develop Reference

SQLite in-memory database encounters SQLITE_LOCKED_SHAREDCACHE intermittently

I am using mybatis 3.4.6 along with org.xerial:sqlite-jdbc 3.28.0. Below is my configuration to use an in-memory database with shared mode enabled
db.driver=org.sqlite.JDBC
db.url=jdbc:sqlite:file::memory:?cache=shared
The db.url is correct according to this test class
And I managed to setup the correct transaction isolation level with below mybatis configuration though there is a typo of property read_uncommitted according to this issue which is reported by me as well
<environment id="${db.env}">
<transactionManager type="jdbc"/>
<dataSource type="POOLED">
<property name="driver" value="${db.driver}" />
<property name="url" value="${db.url}"/>
<property name="username" value="${db.username}" />
<property name="password" value="${db.password}" />
<property name="defaultTransactionIsolationLevel" value="1" />
<property name="driver.synchronous" value="OFF" />
<property name="driver.transaction_mode" value="IMMEDIATE"/>
<property name="driver.foreign_keys" value="ON"/>
</dataSource>
</environment>
This line of configuration
<property name="defaultTransactionIsolationLevel" value="1" />
does the trick to set the correct value of PRAGMA read_uncommitted
I am pretty sure of it since I debugged the underneath code which initialize the connection and check the value has been set correctly
However with the above setting, my program still encounters SQLITE_LOCKED_SHAREDCACHE intermittently while reading, which I think it shouldn't happen according the description highlighted in the red rectangle of below screenshot. I want to know the reason and how to resolve it, though the occurring probability of this error is low.
Any ideas would be appreciated!!
The debug configurations is below
===CONFINGURATION==============================================
jdbcDriver org.sqlite.JDBC
jdbcUrl jdbc:sqlite:file::memory:?cache=shared
jdbcUsername
jdbcPassword ************
poolMaxActiveConnections 10
poolMaxIdleConnections 5
poolMaxCheckoutTime 20000
poolTimeToWait 20000
poolPingEnabled false
poolPingQuery NO PING QUERY SET
poolPingConnectionsNotUsedFor 0
---STATUS-----------------------------------------------------
activeConnections 5
idleConnections 5
requestCount 27
averageRequestTime 7941
averageCheckoutTime 4437
claimedOverdue 0
averageOverdueCheckoutTime 0
hadToWait 0
averageWaitTime 0
badConnectionCount 0
===============================================================
Attachments:
The exception is below
org.apache.ibatis.exceptions.PersistenceException:
### Error querying database. Cause: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit. Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false. Cause: org.sqlite.SQLiteException: [SQLITE_LOCKED_SHAREDCACHE] Contention with a different database connection that shares the cache (database table is locked)
### The error may exist in mapper/MsgRecordDO-sqlmap-mappering.xml
### The error may involve com.super.mock.platform.agent.dal.daointerface.MsgRecordDAO.getRecord
### The error occurred while executing a query
### Cause: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit. Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false. Cause: org.sqlite.SQLiteException: [SQLITE_LOCKED_SHAREDCACHE] Contention with a different database connection that shares the cache (database table is locked)

I finally resolved this issue by myself and share the workaround below in case someone else encounters similar issue in the future.
First of all, we're able to get the completed call stack of the exception shown below
Going through the source code indicated by the callback, we have below findings.
SQLite is built-in with auto commit enabled by default which is contradict with MyBatis which disables auto commit by default since we're using SqlSessionManager
MyBatis would override the auto commit property during connection initialization using method setDesiredAutoCommit which finally invokes SQLiteConnection#setAutoCommit
SQLiteConnection#setAutoCommit would incur a begin immediate operation against the database which is actually exclusive, check out below source code screenshots for detailed explanation since we configure our transaction mode to be IMMEDIATE
<property name="driver.transaction_mode" value="IMMEDIATE"/>
So until now, An apparent solution is to change the transaction mode to be DEFERRED. Furthermore, the solution of making the auto commit setting the same between MyBatis and SQLite has been considered as well, however, it's not adopted since there is no way to set the auto commit of SQLiteConnection during initialization stage, there would be always switching (from true to false or vice versa) and switch would cause the above error probably if transaction mode is not set properly

how to avoid "lock timeout" when updating DB using multiple threads?

I am trying to update a table using multiple threads. But I am not updating the same records/rows at the same time. I am grouping the table into different groups and trying to update them simultaneously. However, I am getting the locked timeout error all the time.
I am using Hibernate, Spring MVC, ThreadPoolTaskExecutor and MySQL. I am getting the data from another DB schema and updating my own database. The data is huge which is why i want to use multi threads so it can be done faster. However, it's producing "lock timeout" error. Can anyone help please? thanks for your good heart.
I call sessionFactory.getCurrenSession() to update the database table.
here is my config:
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close" p:driverClassName="${jdbc.driverClassName}"
p:url="${jdbc.url}" p:username="${jdbc.username}" p:password="${jdbc.password}">
</bean>
<bean id="sessionFactory"
class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="configLocation">
<value>classpath:hibernate.cfg.xml</value>
</property>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.MySQLDialect</prop>
<prop key="hibernate.show_sql">true</prop>
</props>
</property>
</bean>
<bean id="transactionManager"
class="org.springframework.orm.hibernate4.HibernateTransactionManager">
<property name="sessionFactory" ref="sessionFactory" />
</bean>
<bean id="taskExecutor"
class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="5" />
<property name="maxPoolSize" value="10" />
<property name="WaitForTasksToCompleteOnShutdown" value="true" />
</bean>
here is my stacktrace:
WARN : org.hibernate.engine.jdbc.spi.SqlExceptionHelper - SQL Error: 1205, SQLState: 41000
ERROR: org.hibernate.engine.jdbc.spi.SqlExceptionHelper - Lock wait timeout exceeded; try restarting transaction
Exception in thread "taskExecutor-5" Exception in thread "taskExecutor-4" Exception in thread "taskExecutor-2" org.hibernate.exception.LockTimeoutException: could not execute statement
at org.hibernate.dialect.MySQLDialect$1.convert(MySQLDialect.java:407)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:136)
at org.hibernate.hql.internal.ast.exec.BasicExecutor.execute(BasicExecutor.java:103)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.executeUpdate(QueryTranslatorImpl.java:413)
at org.hibernate.engine.query.spi.HQLQueryPlan.performExecuteUpdate(HQLQueryPlan.java:282)
at org.hibernate.internal.SessionImpl.executeUpdate(SessionImpl.java:1289)
at org.hibernate.internal.QueryImpl.executeUpdate(QueryImpl.java:116)
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4232)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4164)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2334)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2262)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2246)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:133)
... 25 more

A "lock wait timeout" can always happen (even with a large amount of inserts in one transaction) and there is no silver bullet to solve it. But I managed to get around it when I was trying to update half of all the records in one (relative small) table while the other half was being modified by another server.
Review all SQL statements in the transaction.
Use explain to make sure indexes are used where possible. Remove any statements that are not needed as part of the transaction.
Optimize the order of the SQL statements in the transaction.
This was a bit of trial and error for me, but try to imagine which order of SQL statements coming from multiple threads/connections might be easier to deal with for the database. In my case, just switching the order of two SQL statements made the "lock wait timeout" occur less frequent.
Update smaller subsets.
This finally solved the "lock wait timeout" for me. In my case there was an indexed column that allowed me to divide the larger update set into smaller subsets. So now one big update transaction was turned into about ten smaller update transactions. Keep in mind though that you need to be able to continue the smaller transactions after a crash (i.e. data must remain consistent in such a way that your application can redo the operation and have the same result).
Whether or not multiple threads will improve the throughput (updated rows per second) remains to be seen: it depends on the size of the update sets (network latency) and how efficiently MySQL can handle the locks for the table(s) to update the rows. You might only see a marginal improvement when using two threads/connections instead of one.
[Edit] Also watch out for database triggers/procedures: they can impact performance in a bad manner.

Maybe you could try to lower isolation level. If it helps you can dig more. It should speed up also execution in multi threaded environment.
If you are using annotations you can achieve this by
#Transactional(isolation=Isolation.READ_UNCOMMITTED)
on top of your transactional class.

This appears to be a timeout on the database side. I'd guess that the database is the limiting factor, so adding threads in your application doesn't help.
If you want to use threads to speed things up, I'd suggest using only two threads. While one thread reads from the other database, the second thread writes to the MySQL database.
Note that if both databases are on the same database server, even that won't help. You would need a faster database or a beefier database machine.

How to trace and prevent the deadlock appeared in c3po which is running in seperate processes?

I have a very simple computation which produces letter matrices finds probably all the words in the matrix. The letters in the word are adjacent cells.
for (int i = 0; i < 500; i++) {
System.out.println(i);
Matrix matrix = new Matrix(4);
matrix.scanWordsRandomly(9);
matrix.printMatrix();
System.out.println(matrix.getSollSize());
matrix.write_to_db();
}
Here is the persisting code.
public void write_to_db() {
Session session = null;
try {
session = HibernateUtil.getSessionFactory().openSession();
session.beginTransaction();
Matrixtr onematrixtr = new Matrixtr();
onematrixtr.setDimension(dimension);
onematrixtr.setMatrixstr(this.toString());
onematrixtr.setSolsize(getSollSize());
session.save(onematrixtr);
for (Map.Entry<Kelimetr, List<Cell>> sollution : sollutions.entrySet()) {
Kelimetr kelimetr = sollution.getKey();
List<Cell> solpath = sollution.getValue();
Solstr onesol = new Solstr();
onesol.setKelimetr(kelimetr);
onesol.setMatrixtr(onematrixtr);
onesol.setSoltext(solpath.toString().replace("[", "").replace("]", "").replace("true", "").replace("false", ""));
session.save(onesol);
}
session.getTransaction().commit();
session.close();
}
catch (HibernateException he) {
System.out.println("DB Error : " + he.getMessage());
session.close();
}
catch (Exception ex) {
System.out.println("General Error : " + ex.getMessage());
}
}
Here is the hibernate configuration file.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-configuration PUBLIC "-//Hibernate/Hibernate Configuration DTD 3.0//EN" "http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="hibernate.dialect">org.hibernate.dialect.MySQLDialect</property>
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.connection.url">jdbc:mysql://localhost:3306/kelimegame_db_dev?autoReconnect=true&useUnicode=true&characterEncoding=UTF-8</property>
<property name="hibernate.connection.username">root</property>
<property name="hibernate.connection.password">!.Wlu9RrCA</property>
<property name="hibernate.show_sql">false</property>
<property name="hibernate.query.factory_class">org.hibernate.hql.classic.ClassicQueryTranslatorFactory</property>
<property name="hibernate.format_sql">false</property>
<!-- Use the C3P0 connection pool provider -->
<property name="hibernate.c3p0.acquire_increment">50</property>
<property name="hibernate.c3p0.min_size">10</property>
<property name="hibernate.c3p0.max_size">100</property>
<property name="hibernate.c3p0.timeout">300</property>
<property name="hibernate.c3p0.max_statements">5</property>
<property name="hibernate.c3p0.idle_test_period">3000</property>
<mapping resource="kelimegame/entity/Progress.hbm.xml"/>
<mapping resource="kelimegame/entity/Solstr.hbm.xml"/>
<mapping resource="kelimegame/entity/Kelimetr.hbm.xml"/>
<mapping resource="kelimegame/entity/User.hbm.xml"/>
<mapping resource="kelimegame/entity/Achievement.hbm.xml"/>
<mapping resource="kelimegame/entity/Matrixtr.hbm.xml"/>
</session-factory>
</hibernate-configuration>
After finding all possible solutions I persist the matrix and the solutions using hibernate. I am also using c3pO library. I am not spawning any thread. All the work is being done in a very simple iterative way. But I am running the jar in separate processes.
From different terminals I am executing this :
java -jar NewDB.jar
I got a deadlock as follows :
Apr 25, 2013 8:38:05 PM com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#7f0c09f9 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Apr 25, 2013 9:08:23 PM com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#7f0c09f9 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#2933f261
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#1
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#116dd369
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#0
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#41529b6f
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#2
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#165ab5ea
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#1d5d211d
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#4d2905fa
Pool thread stack traces:
Thread[C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#1,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:662)
Thread[C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#0,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:662)
Thread[C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#2,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:662)
Apr 25, 2013 9:41:29 PM com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#7f0c09f9 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Apr 25, 2013 9:55:18 PM com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#7f0c09f9 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#5a337b7d
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#0
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#69f079ce
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#1
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#2accf9b8
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#2
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#771eb4fb
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#fc07d6
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#2266731b
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#740f0341
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#59edbee
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#78e924
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#2123aba
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#7acd8a65
Pool thread stack traces:
Thread[C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#0,5,main]
java.text.NumberFormat.getInstance(NumberFormat.java:769)
java.text.NumberFormat.getInstance(NumberFormat.java:393)
java.text.MessageFormat.subformat(MessageFormat.java:1262)
java.text.MessageFormat.format(MessageFormat.java:860)
java.text.Format.format(Format.java:157)
java.text.MessageFormat.format(MessageFormat.java:836)
com.mysql.jdbc.Messages.getString(Messages.java:106)
com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2552)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3002)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2991)
com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3532)
com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:943)
com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4113)
com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1308)
com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2336)
com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2176)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2158)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:525)
com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:183)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:172)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:188)Killed
caglar#ubuntu:~/NetBeansProjects/NewDB/dist$
My question is as follows :
Can this deadlock in c3po happen since I am running the program in
separate processes?
Should I use one process and multiple threads inside of this
process?
How can I trace this deadlock understand the cause of it? Is there a way to trace multiple JVMs causing deadlocks?

this is an interesting one.
you've published two distinct APPARENT DEADLOCKS. the first one is being caused by c3p0 attempting to close() Connections, and those close() operations are neither succeeding nor failing with an Exception in a timely manner. the second APPARENT DEADLOCK shows problems with Connection acquisition: c3p0 is attempting to acquire new Connections, and those attempts are neither succeeding nor failing with an Exception in a timely manner. the fact that very different operations are freezing suggests that it might be a more general problem with your dbms locking up under the stress of what you are doing or somesuch. it should be no problem to run multiple processes against your database, but you need to stay cognizant of limits.
there are a few interesting things about your configuration:
1) hibernate.c3p0.max_statements=5 is a very bad idea, on almost any pool and particularly on pools this large. you've got up to 100 Connections, and you're only allowing a total of 5 Statements to be cached between all of them. this might stress both the pool and the DBMS, as you will constantly be churning through PreparedStatements and the statement cache does a lot of bookkeeping about that. you may have meant that to be 5 cached statements per connection, but that's not what you have configured. you have set a global maximum for your pool. maybe try hibernate.c3p0.maxStatementsPerConnection=5 instead? or set max_statements to zero to turn statement caching off, at least until you resolve your deadlock. see http://www.mchange.com/projects/c3p0/#configuring_statement_pooling
2) if you are running your computation in multiple processes rather than multiple Threads, do you really need each process to hold 50 - 100 Connections? things may well be freezing up simply because you are stressing the dbms with too many Connections outstanding as each of your multiple processes acquire lots of resource-heavy Connections. you don't need more Connections in any process than you might have client Threads running concurrently within that process. i'd set hibernate.c3p0.acquire_increment and probably hibernate.c3p0.max_size to much smaller values.
3) if you really do need all those Connections running simultaneously, you can reduce the vulnerability of your pools to deadlock by increasing the config parameter numHelperThreads to some value greater than its default of 3. you probably want numHelperThreads to be something like twice the number of cores available on your machine. given that you are running multiple processes though, you might find that you are saturating your CPU, and that is freezing things up. so watch for that.
basically, try updating your configuration so that you are using resources -- file handles, network connections, CPU -- as efficiently as possible and so that you are not unnecessarily stressing the pool / statement cache / dbms more than you need to be.
if these suggestions don't resolve the problem, please post the fill config of your pools. c3p0 dumps its config at INFO level on pool initialization.
good luck!

PostgreSQL + Hibernate + C3P0 = FATAL: sorry, too many clients already

I have the following code
Configuration config = new Configuration().configure();
config.buildMappings();
serviceRegistry = new ServiceRegistryBuilder().applySettings(config.getProperties()).buildServiceRegistry();
SessionFactory factory = config.buildSessionFactory(serviceRegistry);
Session hibernateSession = factory.openSession();
Transaction tx = hibernateSession.beginTransaction();
ObjectType ot = (ObjectType)hibernateSession.merge(someObj);
tx.commit();
return ot;
hibernate.cfg.xml contains:
<session-factory>
<property name="connection.url">jdbc:postgresql://127.0.0.1:5432/dbase</property>
<property name="hibernate.dialect">org.hibernate.dialect.PostgreSQLDialect</property>
<property name="connection.driver_class">org.postgresql.Driver</property>
<property name="connection.username">username</property>
<property name="connection.password">password</property>
<property name="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.acquire_increment">1</property>
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">20</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.timeout">300</property>
<property name="hibernate.c3p0.idle_test_period">3000</property>
<property name="hibernate.c3p0.acquireRetryAttempts">1</property>
<property name="hibernate.c3p0.acquireRetryDelay">250</property>
<property name="hibernate.show_sql">true</property>
<property name="hibernate.use_sql_comments">true</property>
<property name="hibernate.transaction.factory_class">org.hibernate.transaction.JDBCTransactionFactory</property>
<property name="hibernate.current_session_context_class">thread</property>
<mapping class="...." />
</session-factory>
After a few seconds and some successful inserts, the following exception appears:
org.postgresql.util.PSQLException: FATAL: sorry, too many clients already
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:291)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)
at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
at org.postgresql.Driver.makeConnection(Driver.java:393)
at org.postgresql.Driver.connect(Driver.java:267)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
12:24:19.151 [ Thread-160] WARN internal.JdbcServicesImpl - HHH000342: Could not obtain connection to query metadata : Connections could not be acquired from the underlying database!
12:24:19.151 [ Thread-160] INFO dialect.Dialect - HHH000400: Using dialect: org.hibernate.dialect.PostgreSQLDialect
12:24:19.151 [ Thread-160] INFO internal.LobCreatorBuilder - HHH000422: Disabling contextual LOB creation as connection was null
12:24:19.151 [ Thread-160] INFO internal.TransactionFactoryInitiator - HHH000268: Transaction strategy: org.hibernate.engine.transaction.internal.jdbc.JdbcTransactionFactory
12:24:19.151 [ Thread-160] INFO ast.ASTQueryTranslatorFactory - HHH000397: Using ASTQueryTranslatorFactory
12:24:19.151 [ Thread-160] INFO hbm2ddl.SchemaUpdate - HHH000228: Running hbm2ddl schema update
12:24:19.151 [ Thread-160] INFO hbm2ddl.SchemaUpdate - HHH000102: Fetching database metadata
12:24:19.211 [Runner$PoolThread-#0] WARN resourcepool.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#ee4084 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (1). Last acquisition attempt exception:
org.postgresql.util.PSQLException: FATAL: sorry, too many clients already
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:291)
It seems that the hibernate doesn't realse the connection. But hibernateSession.close() causes exception Session is closed because tx.commit() is called.

I'm not quite sure what's going on here, but I'd recommend you not set hibernate.c3p0.acquireRetryAttempts to 1. First, that renders your next setting, hibernate.c3p0.acquireRetryDelay irrelevant -- that sets the length of time between retry attempts, but if there is only one attempt (ok, the param name is misleading, it sets the total number of tries), there are no retries. The effect of your settings is simply to have the pool try to fetch a Connection whenever a client comes in, then throw an Exception to clients immediately if that fails. It doesn't at all limit the number of Connections the pool will try to acquire (unless you set breakOnAcquireFailure to true, in which case, with your settings, any failure to acquire a Connection would invalidate the whole pool).
I share sola's concern about your lack of reliable resource cleanup. If, under your settings, commit() means close() (and you are not allowed to call close explicitly? that seems bad), then it is commit that should be in the finally block (but commit in a finally block also seems bad, sometimes you don't want to commit). Whatever the issue with close/commit, with the code you have, occasional Exceptions between openSession and commit will lead to Connection leaks.
But that should not be the cause of your too-many-open-Connections problem. If you leak Connections, you'll find that the Connection pool eventually freezes (as maxPoolSize Connectiosn are checked out forever due to the leaks). You'd only have 25 open Connections. Something else is going on. Try reviewing your logs. Is more than one Connection pool somehow being initialized? (c3p0 dumps config information at INFO level on pool init, so if multiple pools are getting opened, you should see multiple messages. alternatively, you can inspect running c3p0 pools via JMX, to see whether/why more than 25 Connections have been opened.)
Good luck!

I found the cause why c3p0 behaved in this way.The issue was quite trivial...
This part of code:
Configuration config = new Configuration().configure();
config.buildMappings();
serviceRegistry = new ServiceRegistryBuilder().applySettings(config.getProperties()).buildServiceRegistry();
SessionFactory factory = config.buildSessionFactory(serviceRegistry);
was executed multiple times. Thank you Steve for the tip.

I'm suggesting you to use try-catch-finally block,
in finally kindly close the session
i.e
try {
tx.commit();
} catch (HibernateException e) {
handleException(e);
} finally {
hibernateSession.close();
}
and also,
the max_connections property in postgresql.conf it's 100 by default. Increase it if you need.

Spring JDBCTemplate other MySQL datasource than apache commons?

I am using Spring JDBCTemplate to perform SQL operations on an apache commons datasource (org.apache.commons.dbcp.BasicDataSource) and when the service is up and running to long, i end up getting this exception:
org.springframework.dao.RecoverableDataAccessException: StatementCallback; SQL [SELECT * FROM vendor ORDER BY name]; The last packet successfully received from the server was 64,206,061 milliseconds ago. The last packet sent successfully to the server was 64,206,062 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.; nested exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 64,206,061 milliseconds ago. The last packet sent successfully to the server was 64,206,062 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:98)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:406)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:455)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:463)
at com.cable.comcast.neto.nse.taac.dao.VendorDao.getAllVendor(VendorDao.java:25)
at com.cable.comcast.neto.nse.taac.controller.RemoteVendorAccessController.requestAccess(RemoteVendorAccessController.java:78)
I have tried adding the 'autoReconnect=true' to the connection string, but this problem still occurs. Is there another datasource that can be used that will manage the reconnecting for me?

BasicDataSource can manage keeping the connections alive for you. You need to set the following properties :
minEvictableIdleTimeMillis = 120000 // Two minutes
testOnBorrow = true
timeBetweenEvictionRunsMillis = 120000 // Two minutes
minIdle = (some acceptable number of idle connections for your server)
These will configure the data source to keep continually test your connections, and expire and remove them if they become stale. There's a number of other properties on the basic data source that you may want to consider checking into as well to tweak your connection pooling performance. I've run into some strange problems in the past where I was having issues with my database access and it all came down to how the connection pool was configured.

You can try to C3PO:
http://sourceforge.net/projects/c3p0/
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"destroy-method="close">
<property name="user" value="${db.username}"/>
<property name="password" value="${db.password}"/>
<property name="driverClass" value="${db.driverClassName}"/>
<property name="jdbcUrl" value="${db.url}"/>
<property name="initialPoolSize" value="0"/>
<property name="maxPoolSize" value="1"/>
<property name="minPoolSize" value="1"/>
<property name="acquireIncrement" value="1"/>
<property name="acquireRetryAttempts" value="0"/>
<property name="idleConnectionTestPeriod" value="600"/> <!--in seconds-->
</bean>
grettings
pacovr