What is Hibernate's responsibility in regards to database connections it gets from an underlying connection pool. Does it test to see if a connection is closed before it uses it? and if so get another connection from the pool?
I've included error and confirmation info below. Any ideas of where I can start to troubleshoot this would be very helpful. And any advice on the SQL Server driver settings we are using.
from the Catalina log:
04-Nov-2010 21:54:52.691 WARNING org.apache.tomcat.jdbc.pool.ConnectionPool.abandon Connection has been abandoned PooledConnection[ConnectionID:8]:java.lang.Exception
at org.apache.tomcat.jdbc.pool.ConnectionPool.getThreadDump(ConnectionPool.java:926)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:681)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:545)
at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:166)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:106)
from our application log:
2010-11-04 21:54:52,705 [tomcat-http--18] WARN util.JDBCExceptionReporter - SQL Error: 0, SQLState: 08S01
2010-11-04 21:54:52,707 [tomcat-http--18] ERROR util.JDBCExceptionReporter - Socket closed
2010-11-04 21:54:52,708 [tomcat-http--18] ERROR transaction.JDBCTransaction - JDBC rollback failed
java.sql.SQLException: Connection has already been closed.
at org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:112)
at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:94)
at org.apache.tomcat.jdbc.pool.interceptor.AbstractCreateStatementInterceptor.invoke(AbstractCreateStatementInterceptor.java:71)
at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:94)
at org.apache.tomcat.jdbc.pool.interceptor.ConnectionState.invoke(ConnectionState.java:132)
at $Proxy38.rollback(Unknown Source)
at org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:217)
at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:196)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:676)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:845)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:822)
at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:412)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:111)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:625)
The configuration:
<Resource defaultAutoCommit="false" defaultReadOnly="false"
defaultTransactionIsolation="SERIALIZABLE"
driverClassName="com.microsoft.sqlserver.jdbc.SQLServerDriver"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
fairQueue="false" initialSize="10"
jdbcInterceptors="ConnectionState;StatementFinalizer"
jmxEnabled="true" logAbandoned="true" maxActive="100"
maxIdle="10" maxWait="30000"
minEvictableIdleTimeMillis="10000" minIdle="10"
name="com.ourcompany.ap.shoppingcart/datasource"
password="somePassword" removeAbandoned="true"
removeAbandonedTimeout="60" testOnBorrow="true"
testOnReturn="false" testWhileIdle="false"
timeBetweenEvictionRunsMillis="5000"
type="javax.sql.DataSource"
url="jdbc:sqlserver://approd\approd;databaseName=prod"
useEquals="false" username="AccessPointNet"
validationInterval="30000" validationQuery="SELECT 1"/>`
I had a similar problem which was solved by increasing the removeAbandonedTimeout value to a higher number. The problem we faced was due to the query which took longer time that the above mentioned timeout.
What is Hibernate's responsibility in regards to database connections it gets from an underlying connection pool.
Not much, releasing it when the Session gets closed.
Does it test to see if a connection is closed before it uses it? and if so get another connection from the pool?
No, Hibernate doesn't, checking the validity of connection(s) is the responsibility of a connection pool if you want to.
I've included error and confirmation info below. Any ideas of where I can start to troubleshoot this would be very helpful.
What kind of process are you running exactly? A long transaction? Does it timeout? What does the Caused by: say? About the trace:
2010-11-04 21:54:52,705 [tomcat-http--18] WARN util.JDBCExceptionReporter - SQL Error: 0, SQLState: 08S01
2010-11-04 21:54:52,707 [tomcat-http--18] ERROR util.JDBCExceptionReporter - Socket closed
2010-11-04 21:54:52,708 [tomcat-http--18] ERROR transaction.JDBCTransaction - JDBC rollback failed java.sql.SQLException: Connection has already been closed.
Can you reproduce it in a deterministic way? Any networking problem?
And any advice on the SQL Server driver settings we are using.
I've added a great resource about Tomcat and connection pool configuration below. Not specific to SQL Server though.
Resources
Configuring jdbc-pool for high-concurrency
We usually work around this by using dbcp, and providing a validationQuery when definining our data source. Then, dbcp will verify the usability of pooled connections by issuing that query (and transparently recreate the connection should it no longer work), prior to returning them to the application.
Check out
http://tomcat.apache.org/tomcat-6.0-doc/jndi-datasource-examples-howto.html
for more details.
I am currently using liquibase(v1.9) in my project, and when the changeSets run against a blank schema it always takes longer than 60 seconds which results in the thread being marked abandoned I'm not thrilled with increasing the removeAbandonedTimeout value, but this is the only solution I've been able to find to prevent this issue; however, after the initial schema population is complete this is seldom a problem so I set the value back to 60 seconds.
I worked on an issue in the past where we weren't returning connections back to the pool correctly. So, when a connection was used and not returned, making a database call when it was timing out would throw an exception.
We were able to reproduce the issue by making a call to the database, waited 8 hours (postgres' default time out) and tried to make a call to the database again. It throw the same exception every time. Our solution was to rethink (or better yet, add) a connection management strategy.
So, to sum up, are you actually returning your connections to the pool by closing the Session?
I got the solution for the above exception.
Just close the instance of session factory as well while closing the session .
Look at the below Code:
public class HibernateUtil {
private static final SessionFactory sessionFactory = buildSessionFactory();
private static SessionFactory buildSessionFactory() {
try {
// Create the SessionFactory from hibernate.cfg.xml
return new Configuration().configure("hibernate.cfg.xml").buildSessionFactory();
}
catch (Throwable ex) {
ex.printStackTrace();
// Make sure you log the exception, as it might be swallowed
System.err.println("Initial SessionFactory creation failed." + ex);
throw new ExceptionInInitializerError(ex);
}
}
public static SessionFactory getSessionfactory() {
return sessionFactory;
}
public static Session getSession() {
Session session=sessionFactory.openSession();
session.getTransaction().begin();
return session;
}
public static void closeSession(Session session) {
if(session!=null )
{
if(session.getTransaction().isActive())
{
session.getTransaction().commit();
}
session.close();
getSessionfactory().close();
}
}
}
just call the method HibernateUtil.closeSession(). This will solve the problem.
Related
The program runs for a period of time and always reports an error, restarting and returning to normal, the error is as follows：
javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not prepare statementat org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1626)at org.hibernate.query.Query.getResultList(Query.java:165)at net.skycloud.platform.user.repository.KeycloakGroupDao.findAll(KeycloakGroupDao.java:111)at net.skycloud.platform.user.service.UserGroupService.getAllUserGroups(UserGroupService.java:247)at net.skycloud.platform.user.api.rest.UserController.searchUserAndGroup(UserController.java:362)at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.base/java.lang.reflect.Method.invoke(Method.java:566)at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:769)at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1732)at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)at java.base/java.lang.Thread.run(Thread.java:829)Caused by: org.hibernate.exception.GenericJDBCException: could not prepare statementat org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:186)at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:151)at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:2122)at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2059)at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2037)at org.hibernate.loader.Loader.doQuery(Loader.java:956)at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:357)at org.hibernate.loader.Loader.doList(Loader.java:2868)at org.hibernate.loader.Loader.doList(Loader.java:2850)at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2682)at org.hibernate.loader.Loader.list(Loader.java:2677)at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:338)at org.hibernate.internal.SessionImpl.listCustomQuery(SessionImpl.java:2181)at org.hibernate.internal.AbstractSharedSessionContract.list(AbstractSharedSessionContract.java:1204)at org.hibernate.query.internal.NativeQueryImpl.doList(NativeQueryImpl.java:177)at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1617)... 52 common frames omittedCaused by: java.sql.SQLException: Connection is closedat com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:515)at com.sun.proxy.$Proxy156.prepareStatement(Unknown Source)at com.zaxxer.hikari.pool.ProxyConnection.prepareStatement(ProxyConnection.java:337)at com.zaxxer.hikari.pool.HikariProxyConnection.prepareStatement(HikariProxyConnection.java)at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:149)at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:176)... 67 common frames omitted
I hope to fix this bug
Its a database operations level exception using EntityManager or any other session manager (which is not your case) as exception suggest, how ever it can have many reasons such database has closed the connection or network terminated the connection socket, and to solve it you can configure to keep the connection alive but remember you can not hold the hibernate session alive for long time by default .
Also you may have look on OSIV anti pattern.
I have a statement that takes about 20 minutes to run, which is of the form:
create table new_table diststyle key distkey(column1) sortkey(column2)
as (select ....);
When I run it using an SQL IDE or with the psql command line client, the statement executes successfully but when I run it from my Java program, the server closes the connection after 10 minutes with the following exception:
org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [create table new_table diststyle key distkey(column1) sortkey(column2) as (select ....);];
SQL state [HY000]; error code [600001]; [Amazon](600001) The server closed the connection.;
nested exception is java.sql.SQLException: [Amazon](600001) The server closed the connection.
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:419) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:538) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at com.abc.mypackage.MyClass.myMethod(Myclass.java:123) [classes/:?]
Caused by: java.sql.SQLException: [Amazon](600001) The server closed the connection.
at com.amazon.support.channels.TLSSocketChannel.read(Unknown Source) ~[?:?]
Caused by: com.amazon.support.exceptions.GeneralException: [Amazon](600001) The server closed the connection.
at com.amazon.support.channels.TLSSocketChannel.read(Unknown Source) ~[?:?]
I'm using org.apache.commons.dbcp2.BasicDataSource to create connections. I've tried extending the timeout via defaultQueryTimeout, maxConnLifetimeMillis and socketTimeout but to no avail. The server keeps closing the connection after the same 10 minutes.
dataSource = new BasicDataSource();
dataSource.setUsername(dbUser);
dataSource.setPassword(dbPassword);
dataSource.setUrl(dbUrl);
dataSource.setDefaultAutoCommit(true);
dataSource.setTestOnBorrow(true);
dataSource.setTestOnReturn(true);
dataSource.setDriverClassName("com.amazon.redshift.jdbc41.Driver");
dataSource.setDefaultQueryTimeout(7200);
dataSource.setMaxConnLifetimeMillis(7200000);
dataSource.addConnectionProperty("socketTimeout", "7200");
How do I keep the connection alive for longer?
P.S. I do not have any problems establishing connections and running queries that take less than 10 minutes to finish.
You might want to extend your socket timeout.
Current it is 7200ms only:
dataSource.addConnectionProperty("socketTimeout", "7200");
check if the redshift server have a workload management policy that is timing out queries after 10 minutes.
your java code might be setting this policy
You need to set the tcpKeepAlive time to 1 min or less while getting the connection to redshift cluster.
Properties props = new Properties();
props.setProperty("user", user);
props.setProperty("password", password);
props.setProperty("tcpKeepAlive", "true");
props.setProperty("TCPKeepAliveMinutes", "1");
DriverManager.getConnection("jdbc:redshift://"+endpoint+":"
+port+"/"+database, props);
OP here- I was able to make it work by writing wrappers over BasicDataSource and Connection to poll active connection with isValid(int) every few minutes (any frequency more than once-per-10-minutes works). In hindsight, it seems that most timeout-related properties on BasicDataSource apply to connections which are in the pool but are not being used. setDefaultQueryTimeout and tcpKeepAlive + TCPKeepAliveMinutes did not work.
P.S. It has been a while since I resolved this problem and I do not have the code for the wrappers now. Here's a brief description of the wrappers.
WrappedConnection class takes a Connection object (conn) and a TimerTask object (timerTask) in its constructor and implements the Connection interface by simply calling the methods from conn. timerTask calls this.isValid(100) every few minutes as long as the connection is active. WrappedConnection.close stops timerTask and then calls conn.close.
WrappedBasicDataSource implements the DataSource interface, redirecting methods to a BasicDataSource object. BasicDataSourceWrapper.getConnection gets a connection from the aforementioned BasicDataSource and generates a WrappedConnection using the connection and a new TimerTask object.
I might have missed explaining some details but this is the gist of it.
I have a small Java application for testing purposes. I have moved to hikari recently. What I notice is that I keep getting this error.
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:602)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:195)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:85)
Below is my settings for the hikari initially.
HikariConfig config = new HikariConfig();
config.setJdbcUrl("jdbc:mysql://localhost:3306/****");
config.setUsername("***");
config.setPassword("*****");
config.setMaximumPoolSize(20);
Hardly its being used my two devices and I ensure towards the end I do close it. So I don't know why it keep getting the error? What could be the issue or is there some settings which I need to change?
My hikari version is HikariCP-2.6.1.jar.
Your database is not obtaining connection within (30000 milliseconds that is default connectionTimeout property) because of network latency or some of the queries which are taking too long to execute(more than 30000 milliseconds).
Please try to increase value of property connectionTimeout.
YML configuration example:
spring:
datasource:
hikari:
minimumIdle: 2
maximumPoolSize: 10
idleTimeout: 120000
connectionTimeout: 300000
leakDetectionThreshold: 300000
Java Config example:
HikariConfig config = new HikariConfig();
config.setMaximumPoolSize(20);
config.setConnectionTimeout(300000);
config.setConnectionTimeout(120000);
config.setLeakDetectionThreshold(300000);
I am using spring boot and I was facing the same problem, and my solution was to get the connection like this "DataSourceUtils.getConnection(dataSource)". So I change from dataSource.getConnection() to DataSourceUtils.getConnection(dataSource).
In my case the code wasn't closing the connections.
Try-with-resources fixed it:
try (
Connection connection = dataSource.getConnection();
Statement statement = …
) {
…
}
In my case I was using JPA and hence using EntityManagerFactory for persistence and query for my springBoot project and got the same error.
The reason was in any CRUD operation I was not closing EntityManager once the operation is done hence exhausting the resources.
Hope this helps!!
EntityManager em = emf.createEntityManager();
Customer c = em.find(Customer.class , id);
em.close();
request timeout is not something that you can fix by increasing the timeout. Perhaps you'd need to evaluate all the queries from your service and implement indexing if it's needed
This can also happen if the client app is requesting lot of open connections and the database server setting has a max limit on number of pool connections. So the client app is unable to get any more connections from the database server. Check the database server connections pool to see if the max is exceeded during the time period of the errors.
Took forever to figure it out... In my case I used solution similar to #Andres Rincon:
try (Connection connection = DataSourceUtils.getConnection(jdbcTemplate.getDataSource())) {
// some code here
}
What fixed the issue in my case was to add proper indexing in the proper db tables. Take a look at the queries / transactions you're making to the db.
In my case the statement that was causing the latency was an UPDATE statement, e.g.
UPDATE table_name WHERE column1 = value1, column2 = value2;
What fixed the issue for me in this case was to add an index in that table for those two columns like:
CREATE INDEX index_name ON table_name (column1, column2);
Another good reason could be that you're not closing out your connections. You can close the connections with a try-with-resource statement like:
try( Connection connection = datasource.getConnection() ){
//your code
}
In my opinion, increasing the timeout as Girdhar Singh Rathore suggested is not ideal. It could temporarily fix the issue, but at some point you'll need to take care of proper indexing and closing connections management.
Hope this helps.
Generally opened and unclosed connections cause this problem.There is a limit of application servers to connect database and if you over this limit it will be crash your environment.
Connection must be stand on singleton pattern but if you really need to open a datasource or connect external datasource like reports you must close your connection in your finally block where you open connection block
connection.getConnection().rollback();
connection.getConnection().close();
You must also close if you are using PersistenceJpa without singleton
persistenceJPAConfig.dataSource().getConnection().rollback();
persistenceJPAConfig.dataSource().getConnection().close();
If you are using some stress test tools via creating threads to test your methods you probably get this error on your queries which take long time.It will be lead the way optimizing your queries or service instance size.
In my case a:
o.h.engine.jdbc.spi.SqlExceptionHelper: HikariPool-1 - Connection is not available, request timed out after 30019ms.
i.s.commons.web.error.ExceptionLogger: Internal Server Error
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
Was caused by a too low spring.hikari.maximumPoolSize in the application properties, increasing from 5 to 20 solved the issue.
The log message is kind of miss-leading.
In my case I used solution similar to #Andres Rincon:
try (Connection conn = connectionManager.getDataConnection()) {
Statement stmt = conn.createStatement();
...
conn.close();
} catch (Exception e) {
e.printStackTrace();
}
I've fixed my issue using:
increase the minIdle and maxPool
spring.datasource.hikari.minimumIdle=20
spring.datasource.hikari.maximumPoolSize=30
spring.datasource.hikari.connectionTimeout=50000
To debug the issue/check if the values are ok, enable the logging for Hikari:
logging.level.com.zaxxer.hikari.HikariConfig=DEBUG
logging.level.com.zaxxer.hikari=TRACE
The logs will look like:
DEBUG 2023-01-06T16:12:31.932018849Z HikariPool-1 - Before cleanup stats (total=17, active=0, idle=17, waiting=0)
DEBUG 2023-01-06T16:12:31.932665522Z HikariPool-1 - After cleanup stats (total=17, active=0, idle=17, waiting=0)
DEBUG 2023-01-06T16:12:31.932733949Z HikariPool-1 - Fill pool skipped, pool is at sufficient level.
DEBUG 2023-01-06T16:12:32.495269726Z HikariPool-1 - After adding stats (total=17, active=0, idle=17, waiting=0)
DEBUG 2023-01-06T16:12:38.309953158Z HikariPool-1 - Fill pool skipped, pool is at sufficient level.
DEBUG 2023-01-06T16:12:39.200246897Z HikariPool-1 - Fill pool skipped, pool is at sufficient level.
DEBUG 2023-01-06T16:12:44.812065268Z HikariPool-1 - Before cleanup stats (total=18, active=0, idle=18, waiting=0)
DEBUG 2023-01-06T16:12:44.812822113Z HikariPool-1 - After cleanup stats (total=18, active=0, idle=18, waiting=0)
Good Luck ! :)
I'm having issues making a connection to an AS400 database inside of Play!.
My application.conf looks like:
db.default.driver="com.ibm.as400.access.AS400JDBCDriver"
db.default.url="jdbc:as400://SERVER;libraries=A,B,C;toolbox trace=all;trace=true"
db.default.username="user"
db.default.password="password"
I've set up jt400 in the classpath, and I can see under "external libraries" that it shows up and is available. But essentially I get an error message about failing to connect (on user/password I know works) and failure to execute isValid(), which is a function that can not be found inside of AS400JDBCConnection class.
[error] c.z.h.p.PoolBase - HikariPool-1 - Failed to execute isValid() for connection, configure connection test query. (com.ibm.as400.access.AS400JDBCConnection.isValid(I)Z)
[error] application -
! #72265nf0a - Internal server error, for (GET) [/] ->
play.api.Configuration$$anon$1: Configuration error[Cannot connect to database [default]]
at play.api.Configuration$.configError(Configuration.scala:154)
at play.api.Configuration.reportError(Configuration.scala:806)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:48)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:42)
at scala.collection.immutable.List.foreach(List.scala:381)
at play.api.db.DefaultDBApi.connect(DefaultDBApi.scala:42)
at play.api.db.DBApiProvider.get$lzycompute(DBModule.scala:72)
at play.api.db.DBApiProvider.get(DBModule.scala:62)
at play.api.db.DBApiProvider.get(DBModule.scala:58)
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
Caused by: play.api.Configuration$$anon$1: Configuration error[Failed to initialize pool: com.ibm.as400.access.AS400JDBCConnection.isValid(I)Z]
at play.api.Configuration$.configError(Configuration.scala:154)
at play.api.PlayConfig.reportError(Configuration.scala:996)
at play.api.db.HikariCPConnectionPool.create(HikariCPModule.scala:70)
at play.api.db.PooledDatabase.createDataSource(Databases.scala:199)
at play.api.db.DefaultDatabase.dataSource$lzycompute(Databases.scala:123)
at play.api.db.DefaultDatabase.dataSource(Databases.scala:121)
at play.api.db.DefaultDatabase.getConnection(Databases.scala:142)
at play.api.db.DefaultDatabase.getConnection(Databases.scala:138)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:44)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:42)
Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: com.ibm.as400.access.AS400JDBCConnection.isValid(I)Z
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:512)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:105)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:71)
at play.api.db.HikariCPConnectionPool$$anonfun$1.apply(HikariCPModule.scala:58)
at play.api.db.HikariCPConnectionPool$$anonfun$1.apply(HikariCPModule.scala:54)
at scala.util.Try$.apply(Try.scala:192)
at play.api.db.HikariCPConnectionPool.create(HikariCPModule.scala:54)
at play.api.db.PooledDatabase.createDataSource(Databases.scala:199)
at play.api.db.DefaultDatabase.dataSource$lzycompute(Databases.scala:123)
at play.api.db.DefaultDatabase.dataSource(Databases.scala:121)
Caused by: java.lang.AbstractMethodError: com.ibm.as400.access.AS400JDBCConnection.isValid(I)Z
at com.zaxxer.hikari.pool.PoolBase.checkDriverSupport(PoolBase.java:400)
at com.zaxxer.hikari.pool.PoolBase.setupConnection(PoolBase.java:375)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:346)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:506)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:105)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:71)
at play.api.db.HikariCPConnectionPool$$anonfun$1.apply(HikariCPModule.scala:58)
at play.api.db.HikariCPConnectionPool$$anonfun$1.apply(HikariCPModule.scala:54)
at scala.util.Try$.apply(Try.scala:192)
at play.api.db.HikariCPConnectionPool.create(HikariCPModule.scala:54)
I'm able to connect in other java-based projects using something like:
try {
Class.forName("com.ibm.as400.access.AS400JDBCDriver");
Connection con = DriverManager.getConnection("jdbc:as400://" +
ApplicationAuthentication.server + "/" +
ApplicationAuthentication.library,
ApplicationAuthentication.user,
ApplicationAuthentication.password
);
} catch (Exception e) {
System.err.println(e);
throw new WebApplicationException(genericError, Response.Status.UNAUTHORIZED);
}
Guessing from the stacktrace, it appears that the connection returned from your driver is not playing well with the connection Hikari Connection Pool. Hikari is default connection pool in playframework.
Specifically, your exception trace shows that the Hikari CP is attempting to call isValid method on the connection object returned by your JDBC driver and then failing with java.lang.AbstractMethodError.
You can try switching to BoneCP connection pool and see if it helps. You can also check comments on this issue on hikari github issue list
Try adding the following to application.config
db.default.hikaricp.connectionTestQuery="SELECT 1"
Not tested in Play Framework but I had similar issue on spring framework and solved in that way.
Use liquibase datasource like below with connection-test-query
#Bean
#LiquibaseDataSource
public DataSource liquibaseDataSource() {
HikariDataSource dataSource = (HikariDataSource) DataSourceBuilder.create().url("url")
.username("username")
.password("password")
.type(HikariDataSource.class).build();
dataSource.setConnectionTestQuery("select 1 from sysibm.sysdummy1");
return dataSource;
}
Is there any way to determine database connection pool size (connection in used/connection remaining in connection pool) programmatically? We am using Hibernate with C3P0.
We are facing issues while connecting to db. Following exception is thrown and the data is not saved in db.
1005,MA,19/09/11 09:39:14,com.novosys.gtw.business.frontend.SnapshotMessageBusiness.save, Major: Cannot open connection
org.hibernate.exception.GenericJDBCException: Cannot open connection
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:126)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:114)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142)
at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1354)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.hibernate.context.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:342)
at $Proxy0.beginTransaction(Unknown Source)
at com.novosys.gtw.util.base.BaseBusiness.save(BaseBusiness.java:199)
at com.novosys.gtw.business.backend.receivesnapshotmessage.filter.SaveMessageFilter.decode(SaveMessageFilter.java:102)
at org.apache.mina.filter.codec.demux.DemuxingProtocolCodecFactory$ProtocolDecoderImpl.doDecode(DemuxingProtocolCodecFactory.java:292)
at org.apache.mina.filter.codec.CumulativeProtocolDecoder.decode(CumulativeProtocolDecoder.java:133)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:158)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648)
at com.novosys.gtw.business.backend.receivesnapshotmessage.filter.WhitelistFilter.messageReceived(WhitelistFilter.java:231)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648)
at com.novosys.gtw.business.backend.receivesnapshotmessage.filter.MoniterFilter.messageReceived(MoniterFilter.java:92)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java:220)
at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(ExecutorFilter.java:264)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:529)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at org.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:78)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
... 31 more
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1319)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
... 34 more
We tried to resolve it by increasing connection pool size and also increasing no. of connections available at MySQL level, but of no use. We are now trying to sort of debug it to see if its due to connection pool size or due to MySQL connection size. We want to log no. of connection available/in use in connection pool size but could not get any help from google.
Environment: Java, Hibernate, C3P0, MySQL
Session session = null;
Transaction transaction = null;
try {
session = HibernateUtil.getSessionFactory(datasource).getCurrentSession();
transaction = session.beginTransaction();
// db save called here
session.getTransaction().commit();
} catch (Exception e) {
Logger.write(LoggerConstant.MAJOR_ERROR, e.getMessage(), e, methodName);
} finally {
try {
if ((transaction != null) && (transaction.isActive())) {
transaction.rollback();
}
} catch (Exception e) {
Logger.write(LoggerConstant.CRITICAL_ERROR, e.getMessage(), e, methodName);
}
try {
if ((session != null) && (session.isOpen())) {
session.close();
}
} catch (Exception e) {
Logger.write(LoggerConstant.CRITICAL_ERROR, e.getMessage(), e, methodName);
}
}
I don't believe your problem is the connection pool, per se, but more generally a connection leak. This problem is commonly related to the use of HibernateDaoSupport.getSession() without properly pairing with HibernateDaoSupport.releaseSession(). In general, you want something like
public SomeObject getSomething()
{
Session session = null;
try
{
session = this.getSession();
Query query = session.createSQLQuery("SELECT * FROM SomeTable WHERE SomeClause").addEntity(SomeObject.class);
// extract object from query
return someObject;
}
finally
{
if (session != null)
this.releaseSession(session);
}
}
This can be automated by using a HibernateCallback. You do this by providing the query to this.getHibernateTemplate().executeFind which will use a session in Hibernate with automated resource management.
Apart from what ex0du5 has suggested, the exception trace also suggest following:
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1319)
This implies that the connection pool was not able to acquire new connection FROM DATABASE.
Please check the MySQL log for any errors.
Check the maximum connection pool size and max number of connection setting on mysql configuration. (Its least likely that connection pool size will be more that max connection on mysql configuration, but plz make sure of this)
Also there is a way where in you can monitor all the parameters (including max connection setting) of C3P0 conenction pool.
To properly configure the connection pool size, you need to have metrics to investigate the connection usage patterns.
FlexyPool aims to aid you figuring our the right connection pool size, because it can monitor the following metrics:
concurrent connections histogram
concurrent connection requests histogram
data source connection acquiring time histogram
connection lease time histogram
maximum pool size histogram
total connection acquiring time histogram
overflow pool size histogram
retries attempts histogram
You might check the following articles:
FlexyPool, reactive connection pooling
Professional Connection Pool Sizing
The simple scalability equation