Develop Reference

Multi Threaded App With Issues Aquiring DB Connections

I have a Java application that reads data from a CSV file, turns them into business objects, and then saves them to a db. Most of the work is done single-threaded, but some of it is done using multiple threads. The application uses Spring framework, Hibernate as the ORM provider, c3p0 as the connection pool provider, and MySQL as the database. The parts of the application which are multi-threaded are annotated with #Async, using a ThreadPoolTaskExecutor created as a bean for the TaskExecutor that will do this work. The #Aysnc annotated methods return CompletableFuture.
With a small CSV file (under 10k records) everything is fine. However, some of our clients have much larger files (150k records) and this increase in load produces issues during the multi-threaded portions of the application. Here is one of the #Async annotated methods:
public class AppointmentTreeDAO {
#Autowired
private AppointmentRepository appointmentRepository;
private static final Logger logger = LoggerFactory.getLogger(AppointmentTreeDAO.class);
#Async
public CompletableFuture<Appointment> getObjectTree(Long oldApptId) {
logger.info("getting appointment thread");
return CompletableFuture.completedFuture(appointmentRepository.findByApptId(oldApptId));
}
}
This is the AppointmentRepository class (the last method is the one the code in question is calling):
public interface AppointmentRepository extends JpaRepository<Appointment, Long> {
List<Appointment> findByAccountId(String accountId);
List<Appointment> findByAccountIdAndProductIdAndApptStartDatetimeBetween(String accountId,
String productId, Date startDate, Date endDte);
Slice<Appointment> findByAccountIdAndProductIdAndApptStartDatetimeBetween(String accountId,
String productId, Date startDate, Date endDte, Pageable pageable);
#Query(value = "select a.appt_id from appointment a where a.appt_start_datetime " +
"between :startDate AND :endDate and a.account_id = :accountId and a.product_id = :productId",
nativeQuery = true)
List<Long> findByApptIdBetween(#Param("startDate") String startDate, #Param("endDate") String endDate,
#Param("accountId") String accountId, #Param("productId") String productId);
Appointment findByApptId(Long apptId);
}
Since I have that log line in the AppointmentTreeDAO class, I am able to see in the logs that each thread is doing work:
2:58:12.771 PM
2021-11-11 14:58:12.771 INFO 1 --- [mtSave-38] c.t.etl.repository.AppointmentTreeDAO : getting appointment thread
Eventually, I start seeing exceptions like this from the threads (same thread as above in this case):
2:58:16.790 PM
2021-11-11 14:58:16.790 WARN 1 --- [mtSave-38] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: 08001
2:58:16.803 PM
2021-11-11 14:58:16.803 ERROR 1 --- [mtSave-38] o.h.engine.jdbc.spi.SqlExceptionHelper : Could not create connection to database server. Attempted reconnect 3 times. Giving up.
2:58:16.804 PM
2021-11-11 14:58:16.804 WARN 1 --- [mtSave-38] c.t.etl.pipeline.load.Loader : java.util.concurrent.CompletionException: org.springframework.dao.DataAccessResourceFailureException: Unable to acquire JDBC Connection; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.springframework.aop.interceptor.AsyncExecutionAspectSupport.lambda$doSubmit$3(AsyncExecutionAspectSupport.java:279)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.dao.DataAccessResourceFailureException: Unable to acquire JDBC Connection; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:255)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:233)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:551)
at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:61)
at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:242)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:152)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:145)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy148.findByApptId(Unknown Source)
at com.etl.repository.AppointmentTreeDAO.getObjectTree(AppointmentTreeDAO.java:25)
at com.etl.repository.AppointmentTreeDAO$$FastClassBySpringCGLIB$$fb95149d.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at org.springframework.aop.interceptor.AsyncExecutionAspectSupport.lambda$doSubmit$3(AsyncExecutionAspectSupport.java:276)
... 4 more
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:111)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.connection(StatementPreparerImpl.java:50)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:149)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:176)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:151)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:2103)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2040)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2018)
at org.hibernate.loader.Loader.doQuery(Loader.java:948)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349)
at org.hibernate.loader.Loader.doList(Loader.java:2849)
at org.hibernate.loader.Loader.doList(Loader.java:2831)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2663)
at org.hibernate.loader.Loader.list(Loader.java:2658)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:506)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:400)
at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:219)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1414)
at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1625)
at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1593)
at org.hibernate.query.internal.AbstractProducedQuery.getSingleResult(AbstractProducedQuery.java:1641)
at org.hibernate.query.criteria.internal.compile.CriteriaQueryTypeQueryAdapter.getSingleResult(CriteriaQueryTypeQueryAdapter.java:111)
at org.springframework.data.jpa.repository.query.JpaQueryExecution$SingleEntityExecution.doExecute(JpaQueryExecution.java:196)
at org.springframework.data.jpa.repository.query.JpaQueryExecution.execute(JpaQueryExecution.java:88)
at org.springframework.data.jpa.repository.query.AbstractJpaQuery.doExecute(AbstractJpaQuery.java:155)
at org.springframework.data.jpa.repository.query.AbstractJpaQuery.execute(AbstractJpaQuery.java:143)
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:137)
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:121)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:152)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:131)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:80)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
... 22 more
Caused by: java.sql.SQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:73)
at com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:903)
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:828)
at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:453)
at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246)
at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:155)
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:146)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:205)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:169)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108)
... 60 more
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor129.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
at com.mysql.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:89)
at com.mysql.cj.NativeSession.connect(NativeSession.java:144)
at com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:847)
... 73 more
Caused by: java.net.NoRouteToHostException: Cannot assign requested address (Address not available)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.mysql.cj.protocol.StandardSocketFactory.connect(StandardSocketFactory.java:155)
at com.mysql.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:63)
... 75 more
2:58:16.804 PM
2021-11-11 14:58:16.804 WARN 1 --- [mtSave-38] c.t.etl.pipeline.load.Loader : Thread 77 ProcessFutures Loop Exception: org.springframework.dao.DataAccessResourceFailureException: Unable to acquire JDBC Connection;
nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
Soon after throwing the above exception it starts doing work again and its now able to call the db when it couldn't just a short time ago:
2:58:16.807 PM
2021-11-11 14:58:16.807 INFO 1 --- [mtSave-38] c.t.etl.repository.AppointmentTreeDAO : getting appointment thread
This happens intermittently with many different threads, if not all. But eventually all the work is complete and all futures are returned from that method. Then the application returns to being single threaded and the exceptions stop. Once the application returns to using multiple threads the same exceptions as above occur again.
This is my persistence.xml file:
<properties>
<property name="javax.persistence.jdbc.driver" value="com.mysql.jdbc.Driver" />
<property name="hibernate.dialect" value="org.hibernate.dialect.MySQLDialect" />
<property name="hibernate.show_sql" value="false" />
<property name="hibernate.id.new_generator_mappings" value="false" />
<property name="org.hibernate.flushMode" value="ALWAYS" />
<property name="hibernate.c3p0.acquire_increment" value="1" />
<property name="hibernate.c3p0.idle_test_period" value="60"/>
<property name="hibernate.c3p0.max_size" value="54"/>
<property name="hibernate.c3p0.max_statements" value="50"/>
<property name="hibernate.c3p0.min_size" value="5"/>
</properties>
This is how the TaskExecutor is created:
public Executor taskExecutor() {
logger.info("Creating Async Task Executor");
final ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(45);
executor.setMaxPoolSize(45);
executor.setThreadNamePrefix("mtSave-");
executor.initialize();
executor.setKeepAliveSeconds(120);
return executor;
}
I have autoReconnect=true&maxReconnects=10 on my db connection string.
It seems like the threads are being starved for a connection, but I don't understand why. There will always be more work items than there are threads. 150k records of work compared to 45 threads on the high end, and under 10k records of work on the low end. The application never threw these exceptions until multi threading was introduced. I have the connection pool max size (54) set higher than the number of threads I have (45).
Ive tried changing the number of threads, the max size of the connection pool, different size files and did tons of research. Please advise.

Java / JDBC connections aren't thread safe so each thread creates its own connections.
And, MySQL servers limit the number of simultaneous connections they allow. That's a hard limit set at MySQL server startup time. So it's possible your JDBC code in all your threads is attempting to make more connections than the server allows, and so the server rejects them. The app responds with the exceptions you saw.
org.springframework.dao.DataAccessResourceFailureException: Unable to acquire JDBC Connection;
There's another factor to keep in mind. Concurrent queries to the same table (INSERTs or anything else) can contend with one another. Paradoxically, you may get much better throughput with a much smaller number of threads. Try four threads.
If you give the command SHOW GLOBAL STATUS LIKE 'Connection%' you'll see a count of connection errors. Give the command before and after you run your program and take the difference between the values. The values look like this.
Connection_errors_accept errors that occurred during calls to accept() on the listening port.
Connection_errors_internal connections refused due to internal errors in the server, such as failure to start a new thread or an out-of-memory condition.
Connection_errors_max_connections connections refused because the server max_connections limit was reached.
Connection_errors_peer_address errors while searching for connecting client IP addresses.
Connection_errors_select errors during calls to OS select() or poll() on the listening port. (Failure of this operation does not necessarily means a client connection was rejected.)
Connection_errors_tcpwrap connections refused by the libwrap library.
Connections connection attempts (successful or not) to the MySQL server.

How to resolve unable to obtain isolated JDBC connection?

I changed hibernate timeout settings as well as idle time but it not working
My hibernate properties are
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">20</property>
<property name="hibernate.c3p0.timeout">300000</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.idle_test_period">300000</property>
And, I'm getting following exception :
Exception in thread "main" org.hibernate.exception.JDBCConnectionException: unable to obtain isolated JDBC connection
at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:115)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcIsolationDelegate.delegateWork(JdbcIsolationDelegate.java:103)
at org.hibernate.id.enhanced.TableGenerator$1.getNextValue(TableGenerator.java:530)
at org.hibernate.id.enhanced.NoopOptimizer.generate(NoopOptimizer.java:40)
at org.hibernate.id.enhanced.TableGenerator.generate(TableGenerator.java:526)
at org.hibernate.event.internal.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:105)
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.saveWithGeneratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:192)
at org.hibernate.event.internal.DefaultSaveEventListener.saveWithGeneratedOrRequestedId(DefaultSaveEventListener.java:38)
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.entityIsTransient(DefaultSaveOrUpdateEventListener.java:177)
at org.hibernate.event.internal.DefaultSaveEventListener.performSaveOrUpdate(DefaultSaveEventListener.java:32)
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:73)
at org.hibernate.internal.SessionImpl.fireSave(SessionImpl.java:682)
at org.hibernate.internal.SessionImpl.save(SessionImpl.java:674)
at org.hibernate.internal.SessionImpl.save(SessionImpl.java:669)
at com.ihealthmantra.upload.com.ihealthmantra.customer.upload.service.SellerUploadImpl.lambda$4(SellerUploadImpl.java:801)
at java.util.ArrayList.forEach(Unknown Source)
at com.ihealthmantra.upload.com.ihealthmantra.customer.upload.service.SellerUploadImpl.createTimeSlotUploadedSeller(SellerUploadImpl.java:738)
at com.ihealthmantra.upload.com.ihealthmantra.customer.upload.service.SellerUploadImpl.populateSlot(SellerUploadImpl.java:1146)
at com.ihealthmantra.upload.App.main(App.java:81)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 820,677 milliseconds ago. The last packet sent successfully to the server was 820,679 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3746)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2509)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2480)
at com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:4811)
at com.mchange.v2.c3p0.impl.NewProxyConnection.setAutoCommit(NewProxyConnection.java:912)
at org.hibernate.c3p0.internal.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:78)
at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:386)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcIsolationDelegate.delegateWork(JdbcIsolationDelegate.java:48)
... 17 more
Caused by: java.net.SocketException: Connection reset by peer: socket write error
It was working fine earlier, but as data grown recently, it started throwing above exception.

... the server was 820,679 milliseconds ago. is longer than the server
configured value of 'wait_timeout'...
It seems like connection to database server is taking longer time than the value set for timeout. And it was mentioned in your exception as well.
To resolved this issue, either you can increase timeout value or pass autoReconnect=true with your JDBC string (as mentioned in your exception)
To change wait_timeout value in MySQL, you could refer existing post.
MySQL wait_timeout Variable - GLOBAL vs SESSION
OR
Append autoReconnect=true to your JDBC url string
jdbc:mysql://localhost:3306/exodb?autoReconnect=true

JDBC Maximum connection

i am stuck on this problem for the past 2 days, i have a webservice that extract Data from a local database converts it to XML sends to another webservice with HttpGet and gets the response of success or faile and updates my local database. After some time i start to get this Error:
java.sql.SQLException: Couldn't get connection because we are at maximum connection count (20/20) and there are none available
I have tried increasing the maxconnection in my server.xml and applicationContext but nothing changes, always after some time this happens,
This is my datasource
<!-- Data sources -->
<bean id="dataSourceStore" name="dataSourceStore" class="org.springframework.jndi.JndiObjectFactoryBean" destroy-method="close">
<property name="jndiName">
<value>java:comp/env/jdbc/StoreDS</value>
</property>
<property name="removeAbandoned" value="true"/>
<property name="initialSize" value="250" />
<property name="maxActive" value="350" />
</bean>
I have 2 methods that queries de Database, after them i use this one to close
public void close() {
try {
if (!this.getJdbcTemplate().getDataSource().getConnection().isClosed()) {
this.getJdbcTemplate().getDataSource().getConnection().close();
}
} catch (Exception e) {
e.printStackTrace();
// Logger.getLogger(myDAOImpl.class.getName()).log(Level.SEVERE, null, e);
}
}
Heres one example
public void updateStatus(Integer id, String name) {
super.getJdbcTemplate().update(
QueryUtil.QueryAtualizaPedido,
name, id);
close();
}
My server.xml also has maxActive="250", i tried to change this several times but the error always comes as 20/20
The server encountered an internal error () that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>org.hibernate.exception.GenericJDBCException: Cannot open connection org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:126) org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:114) org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66) org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52) org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449) org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167) org.hibernate.jdbc.BorrowedConnectionProxy.invoke(BorrowedConnectionProxy.java:74) com.sun.proxy.$Proxy221.prepareStatement(Unknown Source) br.com.fiorde.dao.conexao.connect.PreparedSt(connect.java:54) br.com.fiorde.servlets.webservice.ImportPedidoXML.validaUsuario(ImportPedidoXML.java:137) br.com.fiorde.servlets.webservice.ImportPedidoXML.doPost(ImportPedidoXML.java:93) br.com.fiorde.servlets.webservice.ImportPedidoXML.doGet(ImportPedidoXML.java:80)javax.servlet.http.HttpServlet.service(HttpServlet.java:617) javax.servlet.http.HttpServlet.service(HttpServlet.java:717) org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)</pre></p><p><b>root cause</b> <pre>java.sql.SQLException: Couldn't get connection because we are at maximum connection count (20/20) and there are none available org.logicalcobwebs.proxool.Prototyper.quickRefuse(Prototyper.java:309) org.logicalcobwebs.proxool.ConnectionPool.getConnection(ConnectionPool.java:152) org.logicalcobwebs.proxool.ProxoolDriver.connect(ProxoolDriver.java:89) java.sql.DriverManager.getConnection(DriverManager.java:571) java.sql.DriverManager.getConnection(DriverManager.java:233) org.hibernate.connection.ProxoolConnectionProvider.getConnection(ProxoolConnectionProvider.java:75) org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446) org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167) org.hibernate.jdbc.BorrowedConnectionProxy.invoke(BorrowedConnectionProxy.java:74) com.sun.proxy.$Proxy221.prepareStatement(Unknown Source) br.com.fiorde.dao.conexao.connect.PreparedSt(connect.java:54) br.com.fiorde.servlets.webservice.ImportPedidoXML.validaUsuario(ImportPedidoXML.java:137) br.com.fiorde.servlets.webservice.ImportPedidoXML.doPost(ImportPedidoXML.java:93) br.com.fiorde.servlets.webservice.ImportPedidoXML.doGet(ImportPedidoXML.java:80) javax.servlet.http.HttpServlet.service(HttpServlet.java:617) javax.servlet.http.HttpServlet.service(HttpServlet.java:717) org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)

This usually is a result of failing to close connections when you are done using them. This is known as a connection leak.
Are you closing connections when you are done using them?

PostgreSQL + Hibernate + C3P0 = FATAL: sorry, too many clients already

I have the following code
Configuration config = new Configuration().configure();
config.buildMappings();
serviceRegistry = new ServiceRegistryBuilder().applySettings(config.getProperties()).buildServiceRegistry();
SessionFactory factory = config.buildSessionFactory(serviceRegistry);
Session hibernateSession = factory.openSession();
Transaction tx = hibernateSession.beginTransaction();
ObjectType ot = (ObjectType)hibernateSession.merge(someObj);
tx.commit();
return ot;
hibernate.cfg.xml contains:
<session-factory>
<property name="connection.url">jdbc:postgresql://127.0.0.1:5432/dbase</property>
<property name="hibernate.dialect">org.hibernate.dialect.PostgreSQLDialect</property>
<property name="connection.driver_class">org.postgresql.Driver</property>
<property name="connection.username">username</property>
<property name="connection.password">password</property>
<property name="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.acquire_increment">1</property>
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">20</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.timeout">300</property>
<property name="hibernate.c3p0.idle_test_period">3000</property>
<property name="hibernate.c3p0.acquireRetryAttempts">1</property>
<property name="hibernate.c3p0.acquireRetryDelay">250</property>
<property name="hibernate.show_sql">true</property>
<property name="hibernate.use_sql_comments">true</property>
<property name="hibernate.transaction.factory_class">org.hibernate.transaction.JDBCTransactionFactory</property>
<property name="hibernate.current_session_context_class">thread</property>
<mapping class="...." />
</session-factory>
After a few seconds and some successful inserts, the following exception appears:
org.postgresql.util.PSQLException: FATAL: sorry, too many clients already
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:291)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)
at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
at org.postgresql.Driver.makeConnection(Driver.java:393)
at org.postgresql.Driver.connect(Driver.java:267)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
12:24:19.151 [ Thread-160] WARN internal.JdbcServicesImpl - HHH000342: Could not obtain connection to query metadata : Connections could not be acquired from the underlying database!
12:24:19.151 [ Thread-160] INFO dialect.Dialect - HHH000400: Using dialect: org.hibernate.dialect.PostgreSQLDialect
12:24:19.151 [ Thread-160] INFO internal.LobCreatorBuilder - HHH000422: Disabling contextual LOB creation as connection was null
12:24:19.151 [ Thread-160] INFO internal.TransactionFactoryInitiator - HHH000268: Transaction strategy: org.hibernate.engine.transaction.internal.jdbc.JdbcTransactionFactory
12:24:19.151 [ Thread-160] INFO ast.ASTQueryTranslatorFactory - HHH000397: Using ASTQueryTranslatorFactory
12:24:19.151 [ Thread-160] INFO hbm2ddl.SchemaUpdate - HHH000228: Running hbm2ddl schema update
12:24:19.151 [ Thread-160] INFO hbm2ddl.SchemaUpdate - HHH000102: Fetching database metadata
12:24:19.211 [Runner$PoolThread-#0] WARN resourcepool.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#ee4084 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (1). Last acquisition attempt exception:
org.postgresql.util.PSQLException: FATAL: sorry, too many clients already
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:291)
It seems that the hibernate doesn't realse the connection. But hibernateSession.close() causes exception Session is closed because tx.commit() is called.

I'm not quite sure what's going on here, but I'd recommend you not set hibernate.c3p0.acquireRetryAttempts to 1. First, that renders your next setting, hibernate.c3p0.acquireRetryDelay irrelevant -- that sets the length of time between retry attempts, but if there is only one attempt (ok, the param name is misleading, it sets the total number of tries), there are no retries. The effect of your settings is simply to have the pool try to fetch a Connection whenever a client comes in, then throw an Exception to clients immediately if that fails. It doesn't at all limit the number of Connections the pool will try to acquire (unless you set breakOnAcquireFailure to true, in which case, with your settings, any failure to acquire a Connection would invalidate the whole pool).
I share sola's concern about your lack of reliable resource cleanup. If, under your settings, commit() means close() (and you are not allowed to call close explicitly? that seems bad), then it is commit that should be in the finally block (but commit in a finally block also seems bad, sometimes you don't want to commit). Whatever the issue with close/commit, with the code you have, occasional Exceptions between openSession and commit will lead to Connection leaks.
But that should not be the cause of your too-many-open-Connections problem. If you leak Connections, you'll find that the Connection pool eventually freezes (as maxPoolSize Connectiosn are checked out forever due to the leaks). You'd only have 25 open Connections. Something else is going on. Try reviewing your logs. Is more than one Connection pool somehow being initialized? (c3p0 dumps config information at INFO level on pool init, so if multiple pools are getting opened, you should see multiple messages. alternatively, you can inspect running c3p0 pools via JMX, to see whether/why more than 25 Connections have been opened.)
Good luck!

I found the cause why c3p0 behaved in this way.The issue was quite trivial...
This part of code:
Configuration config = new Configuration().configure();
config.buildMappings();
serviceRegistry = new ServiceRegistryBuilder().applySettings(config.getProperties()).buildServiceRegistry();
SessionFactory factory = config.buildSessionFactory(serviceRegistry);
was executed multiple times. Thank you Steve for the tip.

I'm suggesting you to use try-catch-finally block,
in finally kindly close the session
i.e
try {
tx.commit();
} catch (HibernateException e) {
handleException(e);
} finally {
hibernateSession.close();
}
and also,
the max_connections property in postgresql.conf it's 100 by default. Increase it if you need.

I am getting Cannot get a connection, pool error Timeout waiting for idle object, When I try to create more than 250 threads in my web application [duplicate]

This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Connection Pool Exception: Cannot get a connection, pool error Timeout waiting for idle object
I am getting Cannot get a connection, pool error Timeout waiting for idle object error, When I try to create more than 250 threads in my web application. I am creating web application using JSF 2.0 and Hibernate.
I have tried with modified hibernate.xml,server.xml,context.xml and also mysql properties.
The followings are I am getting.
WARN (JDBCExceptionReporter.java:233) - SQL Error: 0, SQLState: null
ERROR (JDBCExceptionReporter.java:234) - Cannot get a connection, pool error Timeout waiting for idle object
ERROR (BaseServlet.java:301) - ******** java.lang.Thread.getStackTrace(Thread.java:1426)
Caused by: org.hibernate.exception.GenericJDBCException: Cannot open connection
org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:140)
org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:128)
org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52)
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449)
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:160)
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:81)
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1473)
sun.reflect.GeneratedMethodAccessor378.invoke(Unknown Source)
Caused by: org.apache.tomcat.dbcp.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:92)
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:160)
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:81)
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1473)
sun.reflect.GeneratedMethodAccessor378.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:92)
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:160)
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:81)
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1473)
sun.reflect.GeneratedMethodAccessor378.invoke(Unknown Source)
Please if have any idea about this help me......

Maybe you have set a timeout for "dead" connections and some queries take longer than that. That means that your pool removed a busy connection as "dead" from the pool and requests another from the DB - until the DB pulls the plug.
To debug this, enable logging for your connection pool, so you can see when it requests new connections.
also check your mysql connection settings. and try to close connection when you are done with your db coz next time(beyond maxConnectionAge limit) that connection state will be dead .

I had such issue before, what you need to do is
close hibernate sessions when you are done with it.
e.g.
Session sess = factory.openSession();
Transaction tx;
try {
tx = sess.beginTransaction();
//do some work
...
tx.commit();
}
catch (Exception e) {
if (tx!=null) tx.rollback();
throw e;
}
finally {
sess.close(); // closing session
}