Develop Reference

Getting HikariPool-2 Connection is not available, request timedout

We have a spring boot application with spring JPA and Sybase ASE database. We have used Hikari as the connection pool.
We have 1 stored procedure written which gets called every 4 minute.
Hikari configuration is:
datasource:
        type: com.zaxxer.hikari.HikariDataSource
        data-source-class-name: com.sybase.jdbc4.jdbc.SybDataSource
        driver-class-name: com.sybase.jdbc4.jdbc.SybDriver
        url: xxx-yyy-zzz
        username: xxx-yyy-zzz
        password: xxx-yyy-zzz
        hikari:
            maximum-pool-size: 10
            minimum-idle: 1
The problem is, I am getting below error after every 10 calls to the stored procedure as I have configured pool size 10.
> Caused by: java.sql.SQLTransientConnectionException: HikariPool-2 -
> Connection is not available, request timed out after 30007ms. at
> com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:669)
>
> org.springframework.dao.DataAccessResourceFailureException: Unable to
> acquire JDBC Connection; nested exception is
> org.hibernate.exception.JDBCConnectionException: Unable to acquire
> JDBC Connection
Code is written as follows:
#Autowired
#Lazy
Smsxyz smsxyzImpl;
#PostConstruct
#Scheduled(fixedRate = 240000)
public void completeStatus() {
try {
smsxyzImpl.checkStatus(Constants.STATUS_PER_SCAN, Constants.STATUS_EXPIRY_DURATION, -1L);
} catch (Exception e) {
e.printStackTrace();
}
}
#Autowired
private EntityManager entityManager;
#Override
//#Transactional
public Integer checkStatus(Integer count, Integer expiry, Long userId) {
StoredProcedureQuery query = entityManager.createNamedStoredProcedureQuery("status.check");
query.setParameter("#status_count", count);
query.setParameter("#status_expiry", expiry);
query.setParameter("#user_id", userId);
Integer outputValue= (Integer) query.getOutputParameterValue("#return_status");
return outputValue;
}
If I remove #Transactional then it runs for 10 times (as hikari pool size is 10) and after that it throws an exception as mentioned above
If I keep #Transactional then it runs fine with no exception but it does not update the database.
If I run the stored procedure manually directly from the database, then it works perfectly fine that means no issue in the Stored procedure
My suspect is, it is not releasing the connection. So I closed the connection by entitymanager.close() but it didn't work
I spent many days for this issue but no luck. Could anyone let me know what I am doing wrong here?

Connection not released back to the tomcat jdbc pool

I have a grails2 based application which is using tomcat jdbc pool, recently I have been getting into problem where all the connections in the pool get used up and I start getting:-
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-nio-8443-exec-38] Timeout: Pool empty. Unable to fetch a connection in 10 seconds, none available[size:100; busy:100; idle:0; lastwait:10000].; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-nio-8443-exec-38] Timeout: Pool empty. Unable to fetch a connection in 10 seconds, none available[size:100; busy:100; idle:0; lastwait:10000].
I have a few query that requires heavy join and some stored proc that executes for about 2 - 3 minutes, for it i am manually get the connection from the datasource bean :-
currentConnection = dataSource.connection
sqlInstance = new Sql(currentConnection)
sqlInstance.execute(query)
sqlInstance.close()
I've logged the total active connection in stdout and i see that the no. of active connection keeps on rising and rising and it never drops, it then gets to 100 which is the total active connection allowed and then i start getting issue of poolexhaustauion, can anyone give me an idea, what i might be missing or where the connection might be leaking. here is my connection detail :-
dataSource {
pooled = true
driverClassName = "com.mysql.jdbc.Driver"
url="jdbc:mysql://something:3306/something?zeroDateTimeBehavior=convertToNull&autoReconnect=true&relaxAutoCommit=true"
username="#####"
password='#$#$$$$$$$'
dbCreate = "update"
properties {
initialSize=5
maxActive=100
minIdle=5
maxIdle=25
maxWait = 10000
maxAge = 10 * 60000
timeBetweenEvictionRunsMillis=5000
minEvictableIdleTimeMillis=60000
validationQuery="SELECT 1"
validationInterval=15000
testWhileIdle=true
testOnBorrow=true
testOnReturn=true
removeAbandoned=true
removeAbandonedTimeout=400
logAbandoned=true
jdbcInterceptors = "ConnectionState"
defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
}
}

Redshift server closes connection after 10 minutes

I have a statement that takes about 20 minutes to run, which is of the form:
create table new_table diststyle key distkey(column1) sortkey(column2)
as (select ....);
When I run it using an SQL IDE or with the psql command line client, the statement executes successfully but when I run it from my Java program, the server closes the connection after 10 minutes with the following exception:
org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [create table new_table diststyle key distkey(column1) sortkey(column2) as (select ....);];
SQL state [HY000]; error code [600001]; [Amazon](600001) The server closed the connection.;
nested exception is java.sql.SQLException: [Amazon](600001) The server closed the connection.
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:419) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:538) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at com.abc.mypackage.MyClass.myMethod(Myclass.java:123) [classes/:?]
Caused by: java.sql.SQLException: [Amazon](600001) The server closed the connection.
at com.amazon.support.channels.TLSSocketChannel.read(Unknown Source) ~[?:?]
Caused by: com.amazon.support.exceptions.GeneralException: [Amazon](600001) The server closed the connection.
at com.amazon.support.channels.TLSSocketChannel.read(Unknown Source) ~[?:?]
I'm using org.apache.commons.dbcp2.BasicDataSource to create connections. I've tried extending the timeout via defaultQueryTimeout, maxConnLifetimeMillis and socketTimeout but to no avail. The server keeps closing the connection after the same 10 minutes.
dataSource = new BasicDataSource();
dataSource.setUsername(dbUser);
dataSource.setPassword(dbPassword);
dataSource.setUrl(dbUrl);
dataSource.setDefaultAutoCommit(true);
dataSource.setTestOnBorrow(true);
dataSource.setTestOnReturn(true);
dataSource.setDriverClassName("com.amazon.redshift.jdbc41.Driver");
dataSource.setDefaultQueryTimeout(7200);
dataSource.setMaxConnLifetimeMillis(7200000);
dataSource.addConnectionProperty("socketTimeout", "7200");
How do I keep the connection alive for longer?
P.S. I do not have any problems establishing connections and running queries that take less than 10 minutes to finish.

You might want to extend your socket timeout.
Current it is 7200ms only:
dataSource.addConnectionProperty("socketTimeout", "7200");

check if the redshift server have a workload management policy that is timing out queries after 10 minutes.
your java code might be setting this policy

You need to set the tcpKeepAlive time to 1 min or less while getting the connection to redshift cluster.
Properties props = new Properties();
props.setProperty("user", user);
props.setProperty("password", password);
props.setProperty("tcpKeepAlive", "true");
props.setProperty("TCPKeepAliveMinutes", "1");
DriverManager.getConnection("jdbc:redshift://"+endpoint+":"
+port+"/"+database, props);

OP here- I was able to make it work by writing wrappers over BasicDataSource and Connection to poll active connection with isValid(int) every few minutes (any frequency more than once-per-10-minutes works). In hindsight, it seems that most timeout-related properties on BasicDataSource apply to connections which are in the pool but are not being used. setDefaultQueryTimeout and tcpKeepAlive + TCPKeepAliveMinutes did not work.
P.S. It has been a while since I resolved this problem and I do not have the code for the wrappers now. Here's a brief description of the wrappers.
WrappedConnection class takes a Connection object (conn) and a TimerTask object (timerTask) in its constructor and implements the Connection interface by simply calling the methods from conn. timerTask calls this.isValid(100) every few minutes as long as the connection is active. WrappedConnection.close stops timerTask and then calls conn.close.
WrappedBasicDataSource implements the DataSource interface, redirecting methods to a BasicDataSource object. BasicDataSourceWrapper.getConnection gets a connection from the aforementioned BasicDataSource and generates a WrappedConnection using the connection and a new TimerTask object.
I might have missed explaining some details but this is the gist of it.

HikariCP and Derby

I use the following properties file content:
driverClassName=org.apache.derby.jdbc.EmbeddedDriver
jdbcUrl=jdbc:derby:D:\\development\\databases\\test;create=true
connectionTimeout=3000
and
HikariConfig config = new HikariConfig(propertiesFilePath);
HikariDataSource ds = new HikariDataSource(config);
During
new HikariDataSource(config);
the following exception is thrown:
Caused by:
java.sql.SQLTransientConnectionException: HikariPool-0 - Connection is not available, request timed out after 3002ms.
kari.pool.HikariPool.getConnection(HikariPool.java:195)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:147)
at com.zaxxer.hikari.pool.HikariPool.initializeConnections(HikariPool.java:510)
What I can see is that the database directories are created.
What I am doing wrong?
derby version: 10.10.1.1
HikariCP version: 2.4.3
Update
Well the solution is pretty simple. The connection timeout was just to short.
If the database does not exist, the creation of the db just needs some time.
After increasing the connection timeout everything works fine.

Basics - Troubleshooting Hibernate / JDBC Connection Pool Issue

What is Hibernate's responsibility in regards to database connections it gets from an underlying connection pool. Does it test to see if a connection is closed before it uses it? and if so get another connection from the pool?
I've included error and confirmation info below. Any ideas of where I can start to troubleshoot this would be very helpful. And any advice on the SQL Server driver settings we are using.
from the Catalina log:
04-Nov-2010 21:54:52.691 WARNING org.apache.tomcat.jdbc.pool.ConnectionPool.abandon Connection has been abandoned PooledConnection[ConnectionID:8]:java.lang.Exception
at org.apache.tomcat.jdbc.pool.ConnectionPool.getThreadDump(ConnectionPool.java:926)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:681)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:545)
at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:166)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:106)
from our application log:
2010-11-04 21:54:52,705 [tomcat-http--18] WARN util.JDBCExceptionReporter - SQL Error: 0, SQLState: 08S01
2010-11-04 21:54:52,707 [tomcat-http--18] ERROR util.JDBCExceptionReporter - Socket closed
2010-11-04 21:54:52,708 [tomcat-http--18] ERROR transaction.JDBCTransaction - JDBC rollback failed
java.sql.SQLException: Connection has already been closed.
at org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:112)
at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:94)
at org.apache.tomcat.jdbc.pool.interceptor.AbstractCreateStatementInterceptor.invoke(AbstractCreateStatementInterceptor.java:71)
at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:94)
at org.apache.tomcat.jdbc.pool.interceptor.ConnectionState.invoke(ConnectionState.java:132)
at $Proxy38.rollback(Unknown Source)
at org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:217)
at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:196)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:676)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:845)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:822)
at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:412)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:111)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:625)
The configuration:
<Resource defaultAutoCommit="false" defaultReadOnly="false"
defaultTransactionIsolation="SERIALIZABLE"
driverClassName="com.microsoft.sqlserver.jdbc.SQLServerDriver"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
fairQueue="false" initialSize="10"
jdbcInterceptors="ConnectionState;StatementFinalizer"
jmxEnabled="true" logAbandoned="true" maxActive="100"
maxIdle="10" maxWait="30000"
minEvictableIdleTimeMillis="10000" minIdle="10"
name="com.ourcompany.ap.shoppingcart/datasource"
password="somePassword" removeAbandoned="true"
removeAbandonedTimeout="60" testOnBorrow="true"
testOnReturn="false" testWhileIdle="false"
timeBetweenEvictionRunsMillis="5000"
type="javax.sql.DataSource"
url="jdbc:sqlserver://approd\approd;databaseName=prod"
useEquals="false" username="AccessPointNet"
validationInterval="30000" validationQuery="SELECT 1"/>`

I had a similar problem which was solved by increasing the removeAbandonedTimeout value to a higher number. The problem we faced was due to the query which took longer time that the above mentioned timeout.

What is Hibernate's responsibility in regards to database connections it gets from an underlying connection pool.
Not much, releasing it when the Session gets closed.
Does it test to see if a connection is closed before it uses it? and if so get another connection from the pool?
No, Hibernate doesn't, checking the validity of connection(s) is the responsibility of a connection pool if you want to.
I've included error and confirmation info below. Any ideas of where I can start to troubleshoot this would be very helpful.
What kind of process are you running exactly? A long transaction? Does it timeout? What does the Caused by: say? About the trace:
2010-11-04 21:54:52,705 [tomcat-http--18] WARN util.JDBCExceptionReporter - SQL Error: 0, SQLState: 08S01
2010-11-04 21:54:52,707 [tomcat-http--18] ERROR util.JDBCExceptionReporter - Socket closed
2010-11-04 21:54:52,708 [tomcat-http--18] ERROR transaction.JDBCTransaction - JDBC rollback failed java.sql.SQLException: Connection has already been closed.
Can you reproduce it in a deterministic way? Any networking problem?
And any advice on the SQL Server driver settings we are using.
I've added a great resource about Tomcat and connection pool configuration below. Not specific to SQL Server though.
Resources
Configuring jdbc-pool for high-concurrency

We usually work around this by using dbcp, and providing a validationQuery when definining our data source. Then, dbcp will verify the usability of pooled connections by issuing that query (and transparently recreate the connection should it no longer work), prior to returning them to the application.
Check out
http://tomcat.apache.org/tomcat-6.0-doc/jndi-datasource-examples-howto.html
for more details.

I am currently using liquibase(v1.9) in my project, and when the changeSets run against a blank schema it always takes longer than 60 seconds which results in the thread being marked abandoned I'm not thrilled with increasing the removeAbandonedTimeout value, but this is the only solution I've been able to find to prevent this issue; however, after the initial schema population is complete this is seldom a problem so I set the value back to 60 seconds.

I worked on an issue in the past where we weren't returning connections back to the pool correctly. So, when a connection was used and not returned, making a database call when it was timing out would throw an exception.
We were able to reproduce the issue by making a call to the database, waited 8 hours (postgres' default time out) and tried to make a call to the database again. It throw the same exception every time. Our solution was to rethink (or better yet, add) a connection management strategy.
So, to sum up, are you actually returning your connections to the pool by closing the Session?

I got the solution for the above exception.
Just close the instance of session factory as well while closing the session .
Look at the below Code:
public class HibernateUtil {
private static final SessionFactory sessionFactory = buildSessionFactory();
private static SessionFactory buildSessionFactory() {
try {
// Create the SessionFactory from hibernate.cfg.xml
return new Configuration().configure("hibernate.cfg.xml").buildSessionFactory();
}
catch (Throwable ex) {
ex.printStackTrace();
// Make sure you log the exception, as it might be swallowed
System.err.println("Initial SessionFactory creation failed." + ex);
throw new ExceptionInInitializerError(ex);
}
}
public static SessionFactory getSessionfactory() {
return sessionFactory;
}
public static Session getSession() {
Session session=sessionFactory.openSession();
session.getTransaction().begin();
return session;
}
public static void closeSession(Session session) {
if(session!=null )
{
if(session.getTransaction().isActive())
{
session.getTransaction().commit();
}
session.close();
getSessionfactory().close();
}
}
}
just call the method HibernateUtil.closeSession(). This will solve the problem.