I have Spring boot application, that goes to database using HikariCP.
The thing is, when application gets stop (SIGTERM or other) signal, it must go to database and change some data, after that it closes ("gracefull shutdown"). This logic is written in myShutdownHook (Runtime.getRuntime().addShutdownHook(myShutdownHook)). So the app needs to use connection pool for that. But problem is, that Hikari pool closes right after getting stop signal:
2022-09-18 18:26:44,796 INFO [SpringApplicationShutdownHook] com.zaxxer.hikari.HikariDataSource: HikariPool-1 - Shutdown initiated...
2022-09-18 18:26:44,798 INFO [SpringApplicationShutdownHook] com.zaxxer.hikari.HikariDataSource: HikariPool-1 - Shutdown completed.
And context is closing before my logic is completed. How can I make HikariCP not to close before my logic in myShutdownHook is done?
I currently have an Aws Rds Postgres multi AZ(13.4) database and multiple Java(Spring + hibernate) Applications that use the database with:
<artifactId>postgresql</artifactId>
<version>42.2.24</version>
and
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-hikaricp</artifactId>
<version>5.6.7.Final</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP</artifactId>
<version>4.0.3</version>
<scope>compile</scope>
</dependency>
I configured HikariCP like this:
public DataSource hikariDataSource() {
// Adjusting Cache since RDS failover can/will result in a different IP for the Database
java.security.Security.setProperty("networkaddress.cache.ttl", "1");
java.security.Security.setProperty("networkaddress.cache.negative.ttl", "3");
....
final HikariConfig config = new HikariConfig();
config.setJdbcUrl(jdbcURL);
config.setUsername(user);
config.setPassword(password);
// Initial Connect wait time
config.setConnectionTimeout(60000);
// "We strongly recommend setting this value, and it should be several seconds shorter than any database or
// infrastructure imposed connection time limit."
config.setMaxLifetime(50000);
config.addDataSourceProperty("socketTimeout", "60");
// This value must be less than the maxLifetime value
config.setKeepaliveTime(30000);
config.setMaximumPoolSize(6);
config.setMinimumIdle(2);
config.setIdleTimeout(45000);
return new HikariDataSource(config);
When I trigger a failover of the AWS multi AZ Database HikariCP will acknowledge that the connection is lost and after a while reconnect. So far so good. If my Application was in the middle of a Query while the failover occurred, the Application will throw an error because the Connection was lost, something like:
2022-06-21 10:45:10 WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#7ca90935 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2022-06-21 10:45:10 DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Closing connection org.postgresql.jdbc.PgConnection#7ca90935: (connection is dead)
2022-06-21 10:45:10 DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection#7f52ac10
2022-06-21 10:45:20 WARN com.zaxxer.hikari.pool.ProxyConnection - HikariPool-1 - Connection org.postgresql.jdbc.PgConnection#24118275 marked as broken because of SQLSTATE(08006), ErrorCode(0)
org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:349)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)
...
and
2022-06-21 10:45:20 DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Closing connection org.postgresql.jdbc.PgConnection#24118275: (connection is broken)
2022-06-21 10:45:20 WARN org.hibernate.engine.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: 08006
2022-06-21 10:45:20 ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - An I/O error occurred while sending to the backend.
2022-06-21 10:45:20 WARN org.hibernate.engine.loading.internal.LoadContexts - HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext#314419c8<rs=HikariProxyResultSet#320236451 wrapping org.postgresql.jdbc.PgResultSet#af06e>
2022-06-21 10:45:20 WARN org.hibernate.engine.loading.internal.CollectionLoadContext - HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries
2022-06-21 10:45:20 ERROR org.springframework.transaction.interceptor.TransactionInterceptor - Application exception overridden by rollback exception
org.springframework.dao.DataAccessResourceFailureException: could not extract ResultSet; nested exception is org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:255)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:233)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:551)
at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:61)
...
Now I want to get rid of these errors.
In Mongodb there are Retryable Reads/Writes which wait for a new Primary to be elected, before retrying a query once. Is there something similar I can do with Postgres? I already tried doing a Multi-Az Cluster Setup, but even then the connection is simply lost, error thrown and reestablished.
I'm using Hikari Cp in my spring boot application.
Here is my java configuration file:-
private DataSource buildDataSource(String objectValue) {
HikariDataSource dataSource = new HikariDataSource();
JSONObject obj = new JSONObject(objectValue);
dataSource.setInitializationFailTimeout(0);
dataSource.setMaximumPoolSize(5);
dataSource.setIdleTimeout(10000);
dataSource.setMaxLifetime(45000);
dataSource.setDataSourceClassName(obj.getString("dataSourceClassName"));
dataSource.addDataSourceProperty("url", obj.getString("url"));
dataSource.addDataSourceProperty("user", obj.getString("user"));
dataSource.addDataSourceProperty("password", obj.getString("password"));
return dataSource;
}
when I start the application and after sending first request I'm getting the below logs.
Added connection org.postgresql.jdbc.PgConnection#66c15b95
2021-07-30 12:00:02.788 INFO 17844 --- [nio-8080-exec-1] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2021-07-30 12:00:02.897 DEBUG 17844 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=1, active=0, idle=1, waiting=0)
after few seconds it immediately adds next four connections into idle state:-
Added connection org.postgresql.jdbc.PgConnection#6b9dc322
Added connection org.postgresql.jdbc.PgConnection#6b9dc322
Added connection org.postgresql.jdbc.PgConnection#6b9dc322
Added connection org.postgresql.jdbc.PgConnection#6b9dc322
After adding stats (total=5, active=0, idle=5, waiting=0)
My question here, I sent only a one request and why hikira is adding 4 more additional connections
and already a one idle connection why can't it reuse the same connection.
and I have given the maxLiftTime condition for each connection and after the maxLiftTime passes still the connections are in the idle state.
so any suggestions would be helpful..
From Hikary documentation
🔢minimumIdle: This property controls the minimum number of idle connections that HikariCP tries to maintain in the pool. If the idle connections dip below this value and total connections in the pool are less than maximumPoolSize, HikariCP will make a best effort to add additional connections quickly and efficiently. However, for maximum performance and responsiveness to spike demands, we recommend not setting this value and instead allowing HikariCP to act as a fixed size connection pool. Default: same as maximumPoolSize
It is using as mininimumIdle connection the max pool size you have specified. Set this value to 0 if you do not want any idle connection at all
I have a spring boot app. I'm testing it with testcontainers to ensure that the DB (postgres) and the Repository implementation do what they are supposed to do.
I initialise the container with the following and works pretty well.
#Container
#SuppressWarnings("rawtypes")
private static final PostgreSQLContainer POSTGRE_SQL = new PostgreSQLContainer("postgres:9.6")
.withDatabaseName("xxx")
.withUsername("xxx")
.withPassword("xxx");
static class Initialiser implements ApplicationContextInitializer<ConfigurableApplicationContext> {
#Override
public void initialize(ConfigurableApplicationContext applicationContext) {
TestPropertyValues.of(
"spring.datasource.url=" + POSTGRE_SQL.getJdbcUrl(),
"spring.datasource.username=" + POSTGRE_SQL.getUsername(),
"spring.jpa.hibernate.ddl-auto=create-drop"
).applyTo(applicationContext.getEnvironment());
}
}
The problem is that while the tests are successful, at the end of the class, when the container gets shutdown I get the following error messages from hikari
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#4d728138 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#43070a2e (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#1aa53837 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#3d7cffa2 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#634e7d8e (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#18634db3 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#2bb4ba08 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#71efd133 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#61dd608d (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#6351b7d0 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
They are not making my test failing and i suspect it happens because the container, and therefore the db, is no longer there and hikari still tries to keep the connection pool alive. so the test to complete takes several seconds while hikari officially complain of failing connection.
I tried playing with, settings hikari properties in the Initialiser like "spring.datasource.hikari.maxLifetime=1" and "spring.datasource.hikari.idleTimeout=1" without any luck.
Any suggestions?
Thank you
I had the exact same problem: test passed if I run it individually, but failed when I run it together with other tests.
I found that SpringBootTest is reusing Spring context between tests so there is a common Hikari Pool between tests. But in the background testcontainers killed (after the previous test) a container and created a new one (before the next test). SpringBootTest is not aware of that change resulting in a new Postgres container so Hikari Pool is the same as in the previous test (pointing to already used and currently unavailable port)
In my case adding #DirtiesContext annotation to test helped.
I had the same issue in the question and found that setting the hikari connection timeout of the integration tests helps avoid the delay.
spring:
datasource:
hikari:
connection-timeout: 250
Try other pool
1: Tomcat-jdbc(https://github.com/apache/tomcat/tree/main/modules/jdbc-pool)
2: Druid(https://github.com/alibaba/druid)
3: BeeCP(https://github.com/Chris2018998/BeeCP), which faster than HikariCP
I have a small Java application for testing purposes. I have moved to hikari recently. What I notice is that I keep getting this error.
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:602)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:195)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:85)
Below is my settings for the hikari initially.
HikariConfig config = new HikariConfig();
config.setJdbcUrl("jdbc:mysql://localhost:3306/****");
config.setUsername("***");
config.setPassword("*****");
config.setMaximumPoolSize(20);
Hardly its being used my two devices and I ensure towards the end I do close it. So I don't know why it keep getting the error? What could be the issue or is there some settings which I need to change?
My hikari version is HikariCP-2.6.1.jar.
Your database is not obtaining connection within (30000 milliseconds that is default connectionTimeout property) because of network latency or some of the queries which are taking too long to execute(more than 30000 milliseconds).
Please try to increase value of property connectionTimeout.
YML configuration example:
spring:
datasource:
hikari:
minimumIdle: 2
maximumPoolSize: 10
idleTimeout: 120000
connectionTimeout: 300000
leakDetectionThreshold: 300000
Java Config example:
HikariConfig config = new HikariConfig();
config.setMaximumPoolSize(20);
config.setConnectionTimeout(300000);
config.setConnectionTimeout(120000);
config.setLeakDetectionThreshold(300000);
I am using spring boot and I was facing the same problem, and my solution was to get the connection like this "DataSourceUtils.getConnection(dataSource)". So I change from dataSource.getConnection() to DataSourceUtils.getConnection(dataSource).
In my case the code wasn't closing the connections.
Try-with-resources fixed it:
try (
Connection connection = dataSource.getConnection();
Statement statement = …
) {
…
}
In my case I was using JPA and hence using EntityManagerFactory for persistence and query for my springBoot project and got the same error.
The reason was in any CRUD operation I was not closing EntityManager once the operation is done hence exhausting the resources.
Hope this helps!!
EntityManager em = emf.createEntityManager();
Customer c = em.find(Customer.class , id);
em.close();
request timeout is not something that you can fix by increasing the timeout. Perhaps you'd need to evaluate all the queries from your service and implement indexing if it's needed
This can also happen if the client app is requesting lot of open connections and the database server setting has a max limit on number of pool connections. So the client app is unable to get any more connections from the database server. Check the database server connections pool to see if the max is exceeded during the time period of the errors.
Took forever to figure it out... In my case I used solution similar to #Andres Rincon:
try (Connection connection = DataSourceUtils.getConnection(jdbcTemplate.getDataSource())) {
// some code here
}
What fixed the issue in my case was to add proper indexing in the proper db tables. Take a look at the queries / transactions you're making to the db.
In my case the statement that was causing the latency was an UPDATE statement, e.g.
UPDATE table_name WHERE column1 = value1, column2 = value2;
What fixed the issue for me in this case was to add an index in that table for those two columns like:
CREATE INDEX index_name ON table_name (column1, column2);
Another good reason could be that you're not closing out your connections. You can close the connections with a try-with-resource statement like:
try( Connection connection = datasource.getConnection() ){
//your code
}
In my opinion, increasing the timeout as Girdhar Singh Rathore suggested is not ideal. It could temporarily fix the issue, but at some point you'll need to take care of proper indexing and closing connections management.
Hope this helps.
Generally opened and unclosed connections cause this problem.There is a limit of application servers to connect database and if you over this limit it will be crash your environment.
Connection must be stand on singleton pattern but if you really need to open a datasource or connect external datasource like reports you must close your connection in your finally block where you open connection block
connection.getConnection().rollback();
connection.getConnection().close();
You must also close if you are using PersistenceJpa without singleton
persistenceJPAConfig.dataSource().getConnection().rollback();
persistenceJPAConfig.dataSource().getConnection().close();
If you are using some stress test tools via creating threads to test your methods you probably get this error on your queries which take long time.It will be lead the way optimizing your queries or service instance size.
In my case a:
o.h.engine.jdbc.spi.SqlExceptionHelper: HikariPool-1 - Connection is not available, request timed out after 30019ms.
i.s.commons.web.error.ExceptionLogger: Internal Server Error
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
Was caused by a too low spring.hikari.maximumPoolSize in the application properties, increasing from 5 to 20 solved the issue.
The log message is kind of miss-leading.
In my case I used solution similar to #Andres Rincon:
try (Connection conn = connectionManager.getDataConnection()) {
Statement stmt = conn.createStatement();
...
conn.close();
} catch (Exception e) {
e.printStackTrace();
}
I've fixed my issue using:
increase the minIdle and maxPool
spring.datasource.hikari.minimumIdle=20
spring.datasource.hikari.maximumPoolSize=30
spring.datasource.hikari.connectionTimeout=50000
To debug the issue/check if the values are ok, enable the logging for Hikari:
logging.level.com.zaxxer.hikari.HikariConfig=DEBUG
logging.level.com.zaxxer.hikari=TRACE
The logs will look like:
DEBUG 2023-01-06T16:12:31.932018849Z HikariPool-1 - Before cleanup stats (total=17, active=0, idle=17, waiting=0)
DEBUG 2023-01-06T16:12:31.932665522Z HikariPool-1 - After cleanup stats (total=17, active=0, idle=17, waiting=0)
DEBUG 2023-01-06T16:12:31.932733949Z HikariPool-1 - Fill pool skipped, pool is at sufficient level.
DEBUG 2023-01-06T16:12:32.495269726Z HikariPool-1 - After adding stats (total=17, active=0, idle=17, waiting=0)
DEBUG 2023-01-06T16:12:38.309953158Z HikariPool-1 - Fill pool skipped, pool is at sufficient level.
DEBUG 2023-01-06T16:12:39.200246897Z HikariPool-1 - Fill pool skipped, pool is at sufficient level.
DEBUG 2023-01-06T16:12:44.812065268Z HikariPool-1 - Before cleanup stats (total=18, active=0, idle=18, waiting=0)
DEBUG 2023-01-06T16:12:44.812822113Z HikariPool-1 - After cleanup stats (total=18, active=0, idle=18, waiting=0)
Good Luck ! :)