I currently have an Aws Rds Postgres multi AZ(13.4) database and multiple Java(Spring + hibernate) Applications that use the database with:
<artifactId>postgresql</artifactId>
<version>42.2.24</version>
and
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-hikaricp</artifactId>
<version>5.6.7.Final</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP</artifactId>
<version>4.0.3</version>
<scope>compile</scope>
</dependency>
I configured HikariCP like this:
public DataSource hikariDataSource() {
// Adjusting Cache since RDS failover can/will result in a different IP for the Database
java.security.Security.setProperty("networkaddress.cache.ttl", "1");
java.security.Security.setProperty("networkaddress.cache.negative.ttl", "3");
....
final HikariConfig config = new HikariConfig();
config.setJdbcUrl(jdbcURL);
config.setUsername(user);
config.setPassword(password);
// Initial Connect wait time
config.setConnectionTimeout(60000);
// "We strongly recommend setting this value, and it should be several seconds shorter than any database or
// infrastructure imposed connection time limit."
config.setMaxLifetime(50000);
config.addDataSourceProperty("socketTimeout", "60");
// This value must be less than the maxLifetime value
config.setKeepaliveTime(30000);
config.setMaximumPoolSize(6);
config.setMinimumIdle(2);
config.setIdleTimeout(45000);
return new HikariDataSource(config);
When I trigger a failover of the AWS multi AZ Database HikariCP will acknowledge that the connection is lost and after a while reconnect. So far so good. If my Application was in the middle of a Query while the failover occurred, the Application will throw an error because the Connection was lost, something like:
2022-06-21 10:45:10 WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#7ca90935 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
2022-06-21 10:45:10 DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Closing connection org.postgresql.jdbc.PgConnection#7ca90935: (connection is dead)
2022-06-21 10:45:10 DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection#7f52ac10
2022-06-21 10:45:20 WARN com.zaxxer.hikari.pool.ProxyConnection - HikariPool-1 - Connection org.postgresql.jdbc.PgConnection#24118275 marked as broken because of SQLSTATE(08006), ErrorCode(0)
org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:349)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)
...
and
2022-06-21 10:45:20 DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Closing connection org.postgresql.jdbc.PgConnection#24118275: (connection is broken)
2022-06-21 10:45:20 WARN org.hibernate.engine.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: 08006
2022-06-21 10:45:20 ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - An I/O error occurred while sending to the backend.
2022-06-21 10:45:20 WARN org.hibernate.engine.loading.internal.LoadContexts - HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext#314419c8<rs=HikariProxyResultSet#320236451 wrapping org.postgresql.jdbc.PgResultSet#af06e>
2022-06-21 10:45:20 WARN org.hibernate.engine.loading.internal.CollectionLoadContext - HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries
2022-06-21 10:45:20 ERROR org.springframework.transaction.interceptor.TransactionInterceptor - Application exception overridden by rollback exception
org.springframework.dao.DataAccessResourceFailureException: could not extract ResultSet; nested exception is org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:255)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:233)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:551)
at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:61)
...
Now I want to get rid of these errors.
In Mongodb there are Retryable Reads/Writes which wait for a new Primary to be elected, before retrying a query once. Is there something similar I can do with Postgres? I already tried doing a Multi-Az Cluster Setup, but even then the connection is simply lost, error thrown and reestablished.
We have an application that needs to query 2 MySQL databases (one in each DC). We have noticed that we are constantly closing and opening connections after we query the databases, this seems to be having a hit on performance.
HikariPool-1 - Failed to validate connection com.mysql.cj.jdbc.connectionImpl#xxxx (no operations allowed after connection closed.). Possibly consider using a shorter maxLifetime value.
HikariPool-1 - Closing connection com.mysql.cj.jdbc.connectionImpl#xxxx (connection is dead)
...
HikariPool-1 - Failed to validate connection com.mysql.cj.jdbc.connectionImpl#xxxx (no operations allowed after connection closed.). Possibly consider using a shorter maxLifetime value.
HikariPool-1 - Closing connection com.mysql.cj.jdbc.connectionImpl#xxxx (connection is dead)
...
(repeated per number of connections)
Followed by connections added again and we can see the pool stats confirming this.
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - pool stats (total=23, active=0, idle=23, waiting=0)
...
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - pool stats (total=26, active=0, idle=26, waiting=0)
...
HikariPool-1 - pool stats (total=30, active=0, idle=30, waiting=0)
...
etc
Has anyone seen this before? I have tried changing the maxLifeTime value and idleTimeout with no effect. I have also tried adding connectionTestQuery = SELECT 1, but the issue still happens.
My config:
application-dev.yml
spring:
datasource1:
driverClassName: com.mysql.jdbc.Driver
jdbcUrl: XXXX
password: xxxx
username: xxxx
maximum-pool-size: 40
test-while-idle: true
test-on-borrow: false
connection-timeout: 5000
wait-timeout: 5000
datasource2:
driverClassName: com.mysql.jdbc.Driver
jdbcUrl: XXXX
password: xxxx
username: xxxx
maximum-pool-size: 40
test-while-idle: true
test-on-borrow: false
connection-timeout: 5000
wait-timeout: 5000
We know why the connections where closing. We took the vendors word on the database wait timeout, which they said was MySQL default (28800 seconds (8 hours)) ... it wasn't.
Running:
SHOW VARIABLES LIKE 'wait_%';
Showed the wait_timeout was set to 30 (30seoncds).
For us the fix was to increase the wait_timeout on the db by modifying the config and restarting the db.
We also started to use the hikari keepalivetime.
hikari:
keepaliveTime: 30000
I'm using Hikari Cp in my spring boot application.
Here is my java configuration file:-
private DataSource buildDataSource(String objectValue) {
HikariDataSource dataSource = new HikariDataSource();
JSONObject obj = new JSONObject(objectValue);
dataSource.setInitializationFailTimeout(0);
dataSource.setMaximumPoolSize(5);
dataSource.setIdleTimeout(10000);
dataSource.setMaxLifetime(45000);
dataSource.setDataSourceClassName(obj.getString("dataSourceClassName"));
dataSource.addDataSourceProperty("url", obj.getString("url"));
dataSource.addDataSourceProperty("user", obj.getString("user"));
dataSource.addDataSourceProperty("password", obj.getString("password"));
return dataSource;
}
when I start the application and after sending first request I'm getting the below logs.
Added connection org.postgresql.jdbc.PgConnection#66c15b95
2021-07-30 12:00:02.788 INFO 17844 --- [nio-8080-exec-1] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2021-07-30 12:00:02.897 DEBUG 17844 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=1, active=0, idle=1, waiting=0)
after few seconds it immediately adds next four connections into idle state:-
Added connection org.postgresql.jdbc.PgConnection#6b9dc322
Added connection org.postgresql.jdbc.PgConnection#6b9dc322
Added connection org.postgresql.jdbc.PgConnection#6b9dc322
Added connection org.postgresql.jdbc.PgConnection#6b9dc322
After adding stats (total=5, active=0, idle=5, waiting=0)
My question here, I sent only a one request and why hikira is adding 4 more additional connections
and already a one idle connection why can't it reuse the same connection.
and I have given the maxLiftTime condition for each connection and after the maxLiftTime passes still the connections are in the idle state.
so any suggestions would be helpful..
From Hikary documentation
🔢minimumIdle: This property controls the minimum number of idle connections that HikariCP tries to maintain in the pool. If the idle connections dip below this value and total connections in the pool are less than maximumPoolSize, HikariCP will make a best effort to add additional connections quickly and efficiently. However, for maximum performance and responsiveness to spike demands, we recommend not setting this value and instead allowing HikariCP to act as a fixed size connection pool. Default: same as maximumPoolSize
It is using as mininimumIdle connection the max pool size you have specified. Set this value to 0 if you do not want any idle connection at all
I have a Spring Boot application which uses Hibernate, and HikariDataSource / HikariPool to talk to the database.
A special feature in the app triggers database restart. Currently this breaks the connections in HikariPool:
Caused by: org.postgresql.util.PSQLException: ERROR: relation "relation_which_really_exists" does not exist
Position: 113
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2532)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2267)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:312)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:369)
Old version of the app does call programmatically org.hibernate.SessionFactory.close(); which causes restart of HikariDataSource / HikariCP:
2020-08-17T11:36:42.628Z [qtp1340328248-76] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Shutdown initiated...
2020-08-17T11:36:42.698Z [qtp1340328248-76] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Shutdown completed.
2020-08-17T11:36:51.266Z [qtp1340328248-12] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-2 - Starting...
2020-08-17T11:36:51.515Z [qtp1340328248-12] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-2 - Start completed.
I would like to do the same, but how can I programmatically restart the connection pool in my app? I've seen things like Spring Boot - Handle to Hibernate SessionFactory to get a handle to sessionFactory, and maybe something similar to get a handle to DataSource or CP....but is it OK to close / restart those objects violently, is the Spring Boot context designer to handle such action properly?
There's probably some HikariCP configuration parameters that I can start to experiment with to try reach the same end result, but replicating the old implementation tempts as the easiest / most figured out and most probable to reach same end result.
After some study, found out that you can get handle of HikariCP and trigger connection eviction by:
HikariDataSource hikariDs = (HikariDataSource) dataSource;
HikariPoolMXBean poolBean = hikariDs.getHikariPoolMXBean();
poolBean.softEvictConnections();
There is a parameter in HikariCP which is connectionTestQuery
This is the query that will be executed just before a connection is given to you from the pool to validate that the connection to the database is still alive. So I think you can use it to check if the connection is still alive or not and then force things. But they mention this in the documentation "If your driver supports JDBC4 we strongly recommend not setting this property. This is for "legacy" drivers that do not support the JDBC4 Connection.isValid() API"
There are following possiblties.
make sure allowPoolSuspension is available in configuration.
make sure hikari version 4 or above.
reset credential:-
HikariDataSource ds = (HikariDataSource) applicationContext.getBean("dataSource");
HikariPoolMXBean hikariPoolMXBean = ds.getHikariPoolMXBean();
if (hikariPoolMXBean != null) {
hikariPoolMXBean.softEvictConnections();
}ds.setUsername(userName);ds.setPassword(password);
I have a small Java application for testing purposes. I have moved to hikari recently. What I notice is that I keep getting this error.
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:602)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:195)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:85)
Below is my settings for the hikari initially.
HikariConfig config = new HikariConfig();
config.setJdbcUrl("jdbc:mysql://localhost:3306/****");
config.setUsername("***");
config.setPassword("*****");
config.setMaximumPoolSize(20);
Hardly its being used my two devices and I ensure towards the end I do close it. So I don't know why it keep getting the error? What could be the issue or is there some settings which I need to change?
My hikari version is HikariCP-2.6.1.jar.
Your database is not obtaining connection within (30000 milliseconds that is default connectionTimeout property) because of network latency or some of the queries which are taking too long to execute(more than 30000 milliseconds).
Please try to increase value of property connectionTimeout.
YML configuration example:
spring:
datasource:
hikari:
minimumIdle: 2
maximumPoolSize: 10
idleTimeout: 120000
connectionTimeout: 300000
leakDetectionThreshold: 300000
Java Config example:
HikariConfig config = new HikariConfig();
config.setMaximumPoolSize(20);
config.setConnectionTimeout(300000);
config.setConnectionTimeout(120000);
config.setLeakDetectionThreshold(300000);
I am using spring boot and I was facing the same problem, and my solution was to get the connection like this "DataSourceUtils.getConnection(dataSource)". So I change from dataSource.getConnection() to DataSourceUtils.getConnection(dataSource).
In my case the code wasn't closing the connections.
Try-with-resources fixed it:
try (
Connection connection = dataSource.getConnection();
Statement statement = …
) {
…
}
In my case I was using JPA and hence using EntityManagerFactory for persistence and query for my springBoot project and got the same error.
The reason was in any CRUD operation I was not closing EntityManager once the operation is done hence exhausting the resources.
Hope this helps!!
EntityManager em = emf.createEntityManager();
Customer c = em.find(Customer.class , id);
em.close();
request timeout is not something that you can fix by increasing the timeout. Perhaps you'd need to evaluate all the queries from your service and implement indexing if it's needed
This can also happen if the client app is requesting lot of open connections and the database server setting has a max limit on number of pool connections. So the client app is unable to get any more connections from the database server. Check the database server connections pool to see if the max is exceeded during the time period of the errors.
Took forever to figure it out... In my case I used solution similar to #Andres Rincon:
try (Connection connection = DataSourceUtils.getConnection(jdbcTemplate.getDataSource())) {
// some code here
}
What fixed the issue in my case was to add proper indexing in the proper db tables. Take a look at the queries / transactions you're making to the db.
In my case the statement that was causing the latency was an UPDATE statement, e.g.
UPDATE table_name WHERE column1 = value1, column2 = value2;
What fixed the issue for me in this case was to add an index in that table for those two columns like:
CREATE INDEX index_name ON table_name (column1, column2);
Another good reason could be that you're not closing out your connections. You can close the connections with a try-with-resource statement like:
try( Connection connection = datasource.getConnection() ){
//your code
}
In my opinion, increasing the timeout as Girdhar Singh Rathore suggested is not ideal. It could temporarily fix the issue, but at some point you'll need to take care of proper indexing and closing connections management.
Hope this helps.
Generally opened and unclosed connections cause this problem.There is a limit of application servers to connect database and if you over this limit it will be crash your environment.
Connection must be stand on singleton pattern but if you really need to open a datasource or connect external datasource like reports you must close your connection in your finally block where you open connection block
connection.getConnection().rollback();
connection.getConnection().close();
You must also close if you are using PersistenceJpa without singleton
persistenceJPAConfig.dataSource().getConnection().rollback();
persistenceJPAConfig.dataSource().getConnection().close();
If you are using some stress test tools via creating threads to test your methods you probably get this error on your queries which take long time.It will be lead the way optimizing your queries or service instance size.
In my case a:
o.h.engine.jdbc.spi.SqlExceptionHelper: HikariPool-1 - Connection is not available, request timed out after 30019ms.
i.s.commons.web.error.ExceptionLogger: Internal Server Error
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
Was caused by a too low spring.hikari.maximumPoolSize in the application properties, increasing from 5 to 20 solved the issue.
The log message is kind of miss-leading.
In my case I used solution similar to #Andres Rincon:
try (Connection conn = connectionManager.getDataConnection()) {
Statement stmt = conn.createStatement();
...
conn.close();
} catch (Exception e) {
e.printStackTrace();
}
I've fixed my issue using:
increase the minIdle and maxPool
spring.datasource.hikari.minimumIdle=20
spring.datasource.hikari.maximumPoolSize=30
spring.datasource.hikari.connectionTimeout=50000
To debug the issue/check if the values are ok, enable the logging for Hikari:
logging.level.com.zaxxer.hikari.HikariConfig=DEBUG
logging.level.com.zaxxer.hikari=TRACE
The logs will look like:
DEBUG 2023-01-06T16:12:31.932018849Z HikariPool-1 - Before cleanup stats (total=17, active=0, idle=17, waiting=0)
DEBUG 2023-01-06T16:12:31.932665522Z HikariPool-1 - After cleanup stats (total=17, active=0, idle=17, waiting=0)
DEBUG 2023-01-06T16:12:31.932733949Z HikariPool-1 - Fill pool skipped, pool is at sufficient level.
DEBUG 2023-01-06T16:12:32.495269726Z HikariPool-1 - After adding stats (total=17, active=0, idle=17, waiting=0)
DEBUG 2023-01-06T16:12:38.309953158Z HikariPool-1 - Fill pool skipped, pool is at sufficient level.
DEBUG 2023-01-06T16:12:39.200246897Z HikariPool-1 - Fill pool skipped, pool is at sufficient level.
DEBUG 2023-01-06T16:12:44.812065268Z HikariPool-1 - Before cleanup stats (total=18, active=0, idle=18, waiting=0)
DEBUG 2023-01-06T16:12:44.812822113Z HikariPool-1 - After cleanup stats (total=18, active=0, idle=18, waiting=0)
Good Luck ! :)