I have Spring boot application, that goes to database using HikariCP.
The thing is, when application gets stop (SIGTERM or other) signal, it must go to database and change some data, after that it closes ("gracefull shutdown"). This logic is written in myShutdownHook (Runtime.getRuntime().addShutdownHook(myShutdownHook)). So the app needs to use connection pool for that. But problem is, that Hikari pool closes right after getting stop signal:
2022-09-18 18:26:44,796 INFO [SpringApplicationShutdownHook] com.zaxxer.hikari.HikariDataSource: HikariPool-1 - Shutdown initiated...
2022-09-18 18:26:44,798 INFO [SpringApplicationShutdownHook] com.zaxxer.hikari.HikariDataSource: HikariPool-1 - Shutdown completed.
And context is closing before my logic is completed. How can I make HikariCP not to close before my logic in myShutdownHook is done?
We have an application that needs to query 2 MySQL databases (one in each DC). We have noticed that we are constantly closing and opening connections after we query the databases, this seems to be having a hit on performance.
HikariPool-1 - Failed to validate connection com.mysql.cj.jdbc.connectionImpl#xxxx (no operations allowed after connection closed.). Possibly consider using a shorter maxLifetime value.
HikariPool-1 - Closing connection com.mysql.cj.jdbc.connectionImpl#xxxx (connection is dead)
...
HikariPool-1 - Failed to validate connection com.mysql.cj.jdbc.connectionImpl#xxxx (no operations allowed after connection closed.). Possibly consider using a shorter maxLifetime value.
HikariPool-1 - Closing connection com.mysql.cj.jdbc.connectionImpl#xxxx (connection is dead)
...
(repeated per number of connections)
Followed by connections added again and we can see the pool stats confirming this.
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - pool stats (total=23, active=0, idle=23, waiting=0)
...
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - pool stats (total=26, active=0, idle=26, waiting=0)
...
HikariPool-1 - pool stats (total=30, active=0, idle=30, waiting=0)
...
etc
Has anyone seen this before? I have tried changing the maxLifeTime value and idleTimeout with no effect. I have also tried adding connectionTestQuery = SELECT 1, but the issue still happens.
My config:
application-dev.yml
spring:
datasource1:
driverClassName: com.mysql.jdbc.Driver
jdbcUrl: XXXX
password: xxxx
username: xxxx
maximum-pool-size: 40
test-while-idle: true
test-on-borrow: false
connection-timeout: 5000
wait-timeout: 5000
datasource2:
driverClassName: com.mysql.jdbc.Driver
jdbcUrl: XXXX
password: xxxx
username: xxxx
maximum-pool-size: 40
test-while-idle: true
test-on-borrow: false
connection-timeout: 5000
wait-timeout: 5000
We know why the connections where closing. We took the vendors word on the database wait timeout, which they said was MySQL default (28800 seconds (8 hours)) ... it wasn't.
Running:
SHOW VARIABLES LIKE 'wait_%';
Showed the wait_timeout was set to 30 (30seoncds).
For us the fix was to increase the wait_timeout on the db by modifying the config and restarting the db.
We also started to use the hikari keepalivetime.
hikari:
keepaliveTime: 30000
x and hikari datasource for my jpa queries (version 3.4.5).My problem is I set my datasource maximum pool size property.Even though I set maximum pool size,I see more connection than my maximum poolsize at oracle.At this time when I checked my spring logs I seems connection limit is equal my maximumconnection pool size
Is it possible to hikari datasource can open more connection than my maximum poolsize limit?
In my code side I set maximum connection like that
HikariDataSource dataSource = new HikariDataSource();
dataSource.setMaximumPoolSize(5)
My Oracle Query(I see 7 connections at oracle)
Select * from v$session where where username='xxxxx'
My spring Log(I see 5 connections)
[HikariPool-3 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-3 - Pool stats (total=5, active=0, idle=5, waiting=0)
Yes it can, if HikariPool connectionBag waitingThreadCounts more or equal then maximum pool size.
But in your stacktrace waiting size is 0 and the total is 5 (total value is all connections from connectionBag). So you have a connection in other places.
I have a Spring Boot application which uses Hibernate, and HikariDataSource / HikariPool to talk to the database.
A special feature in the app triggers database restart. Currently this breaks the connections in HikariPool:
Caused by: org.postgresql.util.PSQLException: ERROR: relation "relation_which_really_exists" does not exist
Position: 113
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2532)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2267)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:312)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:369)
Old version of the app does call programmatically org.hibernate.SessionFactory.close(); which causes restart of HikariDataSource / HikariCP:
2020-08-17T11:36:42.628Z [qtp1340328248-76] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Shutdown initiated...
2020-08-17T11:36:42.698Z [qtp1340328248-76] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Shutdown completed.
2020-08-17T11:36:51.266Z [qtp1340328248-12] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-2 - Starting...
2020-08-17T11:36:51.515Z [qtp1340328248-12] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-2 - Start completed.
I would like to do the same, but how can I programmatically restart the connection pool in my app? I've seen things like Spring Boot - Handle to Hibernate SessionFactory to get a handle to sessionFactory, and maybe something similar to get a handle to DataSource or CP....but is it OK to close / restart those objects violently, is the Spring Boot context designer to handle such action properly?
There's probably some HikariCP configuration parameters that I can start to experiment with to try reach the same end result, but replicating the old implementation tempts as the easiest / most figured out and most probable to reach same end result.
After some study, found out that you can get handle of HikariCP and trigger connection eviction by:
HikariDataSource hikariDs = (HikariDataSource) dataSource;
HikariPoolMXBean poolBean = hikariDs.getHikariPoolMXBean();
poolBean.softEvictConnections();
There is a parameter in HikariCP which is connectionTestQuery
This is the query that will be executed just before a connection is given to you from the pool to validate that the connection to the database is still alive. So I think you can use it to check if the connection is still alive or not and then force things. But they mention this in the documentation "If your driver supports JDBC4 we strongly recommend not setting this property. This is for "legacy" drivers that do not support the JDBC4 Connection.isValid() API"
There are following possiblties.
make sure allowPoolSuspension is available in configuration.
make sure hikari version 4 or above.
reset credential:-
HikariDataSource ds = (HikariDataSource) applicationContext.getBean("dataSource");
HikariPoolMXBean hikariPoolMXBean = ds.getHikariPoolMXBean();
if (hikariPoolMXBean != null) {
hikariPoolMXBean.softEvictConnections();
}ds.setUsername(userName);ds.setPassword(password);
I have a small Java application for testing purposes. I have moved to hikari recently. What I notice is that I keep getting this error.
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:602)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:195)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:85)
Below is my settings for the hikari initially.
HikariConfig config = new HikariConfig();
config.setJdbcUrl("jdbc:mysql://localhost:3306/****");
config.setUsername("***");
config.setPassword("*****");
config.setMaximumPoolSize(20);
Hardly its being used my two devices and I ensure towards the end I do close it. So I don't know why it keep getting the error? What could be the issue or is there some settings which I need to change?
My hikari version is HikariCP-2.6.1.jar.
Your database is not obtaining connection within (30000 milliseconds that is default connectionTimeout property) because of network latency or some of the queries which are taking too long to execute(more than 30000 milliseconds).
Please try to increase value of property connectionTimeout.
YML configuration example:
spring:
datasource:
hikari:
minimumIdle: 2
maximumPoolSize: 10
idleTimeout: 120000
connectionTimeout: 300000
leakDetectionThreshold: 300000
Java Config example:
HikariConfig config = new HikariConfig();
config.setMaximumPoolSize(20);
config.setConnectionTimeout(300000);
config.setConnectionTimeout(120000);
config.setLeakDetectionThreshold(300000);
I am using spring boot and I was facing the same problem, and my solution was to get the connection like this "DataSourceUtils.getConnection(dataSource)". So I change from dataSource.getConnection() to DataSourceUtils.getConnection(dataSource).
In my case the code wasn't closing the connections.
Try-with-resources fixed it:
try (
Connection connection = dataSource.getConnection();
Statement statement = …
) {
…
}
In my case I was using JPA and hence using EntityManagerFactory for persistence and query for my springBoot project and got the same error.
The reason was in any CRUD operation I was not closing EntityManager once the operation is done hence exhausting the resources.
Hope this helps!!
EntityManager em = emf.createEntityManager();
Customer c = em.find(Customer.class , id);
em.close();
request timeout is not something that you can fix by increasing the timeout. Perhaps you'd need to evaluate all the queries from your service and implement indexing if it's needed
This can also happen if the client app is requesting lot of open connections and the database server setting has a max limit on number of pool connections. So the client app is unable to get any more connections from the database server. Check the database server connections pool to see if the max is exceeded during the time period of the errors.
Took forever to figure it out... In my case I used solution similar to #Andres Rincon:
try (Connection connection = DataSourceUtils.getConnection(jdbcTemplate.getDataSource())) {
// some code here
}
What fixed the issue in my case was to add proper indexing in the proper db tables. Take a look at the queries / transactions you're making to the db.
In my case the statement that was causing the latency was an UPDATE statement, e.g.
UPDATE table_name WHERE column1 = value1, column2 = value2;
What fixed the issue for me in this case was to add an index in that table for those two columns like:
CREATE INDEX index_name ON table_name (column1, column2);
Another good reason could be that you're not closing out your connections. You can close the connections with a try-with-resource statement like:
try( Connection connection = datasource.getConnection() ){
//your code
}
In my opinion, increasing the timeout as Girdhar Singh Rathore suggested is not ideal. It could temporarily fix the issue, but at some point you'll need to take care of proper indexing and closing connections management.
Hope this helps.
Generally opened and unclosed connections cause this problem.There is a limit of application servers to connect database and if you over this limit it will be crash your environment.
Connection must be stand on singleton pattern but if you really need to open a datasource or connect external datasource like reports you must close your connection in your finally block where you open connection block
connection.getConnection().rollback();
connection.getConnection().close();
You must also close if you are using PersistenceJpa without singleton
persistenceJPAConfig.dataSource().getConnection().rollback();
persistenceJPAConfig.dataSource().getConnection().close();
If you are using some stress test tools via creating threads to test your methods you probably get this error on your queries which take long time.It will be lead the way optimizing your queries or service instance size.
In my case a:
o.h.engine.jdbc.spi.SqlExceptionHelper: HikariPool-1 - Connection is not available, request timed out after 30019ms.
i.s.commons.web.error.ExceptionLogger: Internal Server Error
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
Was caused by a too low spring.hikari.maximumPoolSize in the application properties, increasing from 5 to 20 solved the issue.
The log message is kind of miss-leading.
In my case I used solution similar to #Andres Rincon:
try (Connection conn = connectionManager.getDataConnection()) {
Statement stmt = conn.createStatement();
...
conn.close();
} catch (Exception e) {
e.printStackTrace();
}
I've fixed my issue using:
increase the minIdle and maxPool
spring.datasource.hikari.minimumIdle=20
spring.datasource.hikari.maximumPoolSize=30
spring.datasource.hikari.connectionTimeout=50000
To debug the issue/check if the values are ok, enable the logging for Hikari:
logging.level.com.zaxxer.hikari.HikariConfig=DEBUG
logging.level.com.zaxxer.hikari=TRACE
The logs will look like:
DEBUG 2023-01-06T16:12:31.932018849Z HikariPool-1 - Before cleanup stats (total=17, active=0, idle=17, waiting=0)
DEBUG 2023-01-06T16:12:31.932665522Z HikariPool-1 - After cleanup stats (total=17, active=0, idle=17, waiting=0)
DEBUG 2023-01-06T16:12:31.932733949Z HikariPool-1 - Fill pool skipped, pool is at sufficient level.
DEBUG 2023-01-06T16:12:32.495269726Z HikariPool-1 - After adding stats (total=17, active=0, idle=17, waiting=0)
DEBUG 2023-01-06T16:12:38.309953158Z HikariPool-1 - Fill pool skipped, pool is at sufficient level.
DEBUG 2023-01-06T16:12:39.200246897Z HikariPool-1 - Fill pool skipped, pool is at sufficient level.
DEBUG 2023-01-06T16:12:44.812065268Z HikariPool-1 - Before cleanup stats (total=18, active=0, idle=18, waiting=0)
DEBUG 2023-01-06T16:12:44.812822113Z HikariPool-1 - After cleanup stats (total=18, active=0, idle=18, waiting=0)
Good Luck ! :)