Develop Reference

Spring boot & mysql - connections closing

We have an application that needs to query 2 MySQL databases (one in each DC). We have noticed that we are constantly closing and opening connections after we query the databases, this seems to be having a hit on performance.
HikariPool-1 - Failed to validate connection com.mysql.cj.jdbc.connectionImpl#xxxx (no operations allowed after connection closed.). Possibly consider using a shorter maxLifetime value.
HikariPool-1 - Closing connection com.mysql.cj.jdbc.connectionImpl#xxxx (connection is dead)
...
HikariPool-1 - Failed to validate connection com.mysql.cj.jdbc.connectionImpl#xxxx (no operations allowed after connection closed.). Possibly consider using a shorter maxLifetime value.
HikariPool-1 - Closing connection com.mysql.cj.jdbc.connectionImpl#xxxx (connection is dead)
...
(repeated per number of connections)
Followed by connections added again and we can see the pool stats confirming this.
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - pool stats (total=23, active=0, idle=23, waiting=0)
...
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - Added connection com.mysql.cj.jdbc.connectionImpl#xxxx
HikariPool-1 - pool stats (total=26, active=0, idle=26, waiting=0)
...
HikariPool-1 - pool stats (total=30, active=0, idle=30, waiting=0)
...
etc
Has anyone seen this before? I have tried changing the maxLifeTime value and idleTimeout with no effect. I have also tried adding connectionTestQuery = SELECT 1, but the issue still happens.
My config:
application-dev.yml
spring:
datasource1:
driverClassName: com.mysql.jdbc.Driver
jdbcUrl: XXXX
password: xxxx
username: xxxx
maximum-pool-size: 40
test-while-idle: true
test-on-borrow: false
connection-timeout: 5000
wait-timeout: 5000
datasource2:
driverClassName: com.mysql.jdbc.Driver
jdbcUrl: XXXX
password: xxxx
username: xxxx
maximum-pool-size: 40
test-while-idle: true
test-on-borrow: false
connection-timeout: 5000
wait-timeout: 5000

We know why the connections where closing. We took the vendors word on the database wait timeout, which they said was MySQL default (28800 seconds (8 hours)) ... it wasn't.
Running:
SHOW VARIABLES LIKE 'wait_%';
Showed the wait_timeout was set to 30 (30seoncds).
For us the fix was to increase the wait_timeout on the db by modifying the config and restarting the db.
We also started to use the hikari keepalivetime.
hikari:
keepaliveTime: 30000

Hikari connection pool postgres

I'm using Hikari Cp in my spring boot application.
Here is my java configuration file:-
private DataSource buildDataSource(String objectValue) {
HikariDataSource dataSource = new HikariDataSource();
JSONObject obj = new JSONObject(objectValue);
dataSource.setInitializationFailTimeout(0);
dataSource.setMaximumPoolSize(5);
dataSource.setIdleTimeout(10000);
dataSource.setMaxLifetime(45000);
dataSource.setDataSourceClassName(obj.getString("dataSourceClassName"));
dataSource.addDataSourceProperty("url", obj.getString("url"));
dataSource.addDataSourceProperty("user", obj.getString("user"));
dataSource.addDataSourceProperty("password", obj.getString("password"));
return dataSource;
}
when I start the application and after sending first request I'm getting the below logs.
Added connection org.postgresql.jdbc.PgConnection#66c15b95
2021-07-30 12:00:02.788 INFO 17844 --- [nio-8080-exec-1] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2021-07-30 12:00:02.897 DEBUG 17844 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=1, active=0, idle=1, waiting=0)
after few seconds it immediately adds next four connections into idle state:-
Added connection org.postgresql.jdbc.PgConnection#6b9dc322
Added connection org.postgresql.jdbc.PgConnection#6b9dc322
Added connection org.postgresql.jdbc.PgConnection#6b9dc322
Added connection org.postgresql.jdbc.PgConnection#6b9dc322
After adding stats (total=5, active=0, idle=5, waiting=0)
My question here, I sent only a one request and why hikira is adding 4 more additional connections
and already a one idle connection why can't it reuse the same connection.
and I have given the maxLiftTime condition for each connection and after the maxLiftTime passes still the connections are in the idle state.
so any suggestions would be helpful..

From Hikary documentation
🔢minimumIdle: This property controls the minimum number of idle connections that HikariCP tries to maintain in the pool. If the idle connections dip below this value and total connections in the pool are less than maximumPoolSize, HikariCP will make a best effort to add additional connections quickly and efficiently. However, for maximum performance and responsiveness to spike demands, we recommend not setting this value and instead allowing HikariCP to act as a fixed size connection pool. Default: same as maximumPoolSize
It is using as mininimumIdle connection the max pool size you have specified. Set this value to 0 if you do not want any idle connection at all

Programmatically restart HikariPool in Spring Boot application?

I have a Spring Boot application which uses Hibernate, and HikariDataSource / HikariPool to talk to the database.
A special feature in the app triggers database restart. Currently this breaks the connections in HikariPool:
Caused by: org.postgresql.util.PSQLException: ERROR: relation "relation_which_really_exists" does not exist
Position: 113
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2532)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2267)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:312)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:369)
Old version of the app does call programmatically org.hibernate.SessionFactory.close(); which causes restart of HikariDataSource / HikariCP:
2020-08-17T11:36:42.628Z [qtp1340328248-76] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Shutdown initiated...
2020-08-17T11:36:42.698Z [qtp1340328248-76] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Shutdown completed.
2020-08-17T11:36:51.266Z [qtp1340328248-12] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-2 - Starting...
2020-08-17T11:36:51.515Z [qtp1340328248-12] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-2 - Start completed.
I would like to do the same, but how can I programmatically restart the connection pool in my app? I've seen things like Spring Boot - Handle to Hibernate SessionFactory to get a handle to sessionFactory, and maybe something similar to get a handle to DataSource or CP....but is it OK to close / restart those objects violently, is the Spring Boot context designer to handle such action properly?
There's probably some HikariCP configuration parameters that I can start to experiment with to try reach the same end result, but replicating the old implementation tempts as the easiest / most figured out and most probable to reach same end result.

After some study, found out that you can get handle of HikariCP and trigger connection eviction by:
HikariDataSource hikariDs = (HikariDataSource) dataSource;
HikariPoolMXBean poolBean = hikariDs.getHikariPoolMXBean();
poolBean.softEvictConnections();

There is a parameter in HikariCP which is connectionTestQuery
This is the query that will be executed just before a connection is given to you from the pool to validate that the connection to the database is still alive. So I think you can use it to check if the connection is still alive or not and then force things. But they mention this in the documentation "If your driver supports JDBC4 we strongly recommend not setting this property. This is for "legacy" drivers that do not support the JDBC4 Connection.isValid() API"

There are following possiblties.
make sure allowPoolSuspension is available in configuration.
make sure hikari version 4 or above.
reset credential:-
HikariDataSource ds = (HikariDataSource) applicationContext.getBean("dataSource");
HikariPoolMXBean hikariPoolMXBean = ds.getHikariPoolMXBean();
if (hikariPoolMXBean != null) {
hikariPoolMXBean.softEvictConnections();
}ds.setUsername(userName);ds.setPassword(password);

testcontainers, hikari and Failed to validate connection org.postgresql.jdbc.PgConnection

I have a spring boot app. I'm testing it with testcontainers to ensure that the DB (postgres) and the Repository implementation do what they are supposed to do.
I initialise the container with the following and works pretty well.
#Container
#SuppressWarnings("rawtypes")
private static final PostgreSQLContainer POSTGRE_SQL = new PostgreSQLContainer("postgres:9.6")
.withDatabaseName("xxx")
.withUsername("xxx")
.withPassword("xxx");
static class Initialiser implements ApplicationContextInitializer<ConfigurableApplicationContext> {
#Override
public void initialize(ConfigurableApplicationContext applicationContext) {
TestPropertyValues.of(
"spring.datasource.url=" + POSTGRE_SQL.getJdbcUrl(),
"spring.datasource.username=" + POSTGRE_SQL.getUsername(),
"spring.jpa.hibernate.ddl-auto=create-drop"
).applyTo(applicationContext.getEnvironment());
}
}
The problem is that while the tests are successful, at the end of the class, when the container gets shutdown I get the following error messages from hikari
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#4d728138 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#43070a2e (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#1aa53837 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#3d7cffa2 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#634e7d8e (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#18634db3 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#2bb4ba08 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#71efd133 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#61dd608d (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
[31mWARN [0;39m [36mcom.zaxxer.hikari.pool.PoolBase.isConnectionAlive[0;39m - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#6351b7d0 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
They are not making my test failing and i suspect it happens because the container, and therefore the db, is no longer there and hikari still tries to keep the connection pool alive. so the test to complete takes several seconds while hikari officially complain of failing connection.
I tried playing with, settings hikari properties in the Initialiser like "spring.datasource.hikari.maxLifetime=1" and "spring.datasource.hikari.idleTimeout=1" without any luck.
Any suggestions?
Thank you

I had the exact same problem: test passed if I run it individually, but failed when I run it together with other tests.
I found that SpringBootTest is reusing Spring context between tests so there is a common Hikari Pool between tests. But in the background testcontainers killed (after the previous test) a container and created a new one (before the next test). SpringBootTest is not aware of that change resulting in a new Postgres container so Hikari Pool is the same as in the previous test (pointing to already used and currently unavailable port)
In my case adding #DirtiesContext annotation to test helped.

I had the same issue in the question and found that setting the hikari connection timeout of the integration tests helps avoid the delay.
spring:
datasource:
hikari:
connection-timeout: 250

Try other pool
1: Tomcat-jdbc(https://github.com/apache/tomcat/tree/main/modules/jdbc-pool)
2: Druid(https://github.com/alibaba/druid)
3: BeeCP(https://github.com/Chris2018998/BeeCP), which faster than HikariCP

why does spring-boot and postgres connection breaks after certain time?

I run Spring-Boot with gradle using the tomcat-connection-pool. All the standard spring-boot-tools. I run several soap-webservices on that webserver. It all works fine when testing the load of the server. But after a doing nothing for ~7.5hours this exception occurs. Sure its a timeout but I try to prevent it with the following:
spring.datasource.url=jdbc:postgresql://mydb?autoReconnect=true
I use #transactional for certain statements. But in general I only use the JPA-Repository from spring-boot.
The connections are managed by the tomcat-connection-pool so there is no idle-connection-problem.
When I restart the application-server everything runs fine again.
My Database-server runs PostgreSQL 9.4.1 on x86_64-unknown-linux-gnu and there is NO firewall between the database and the app-server.
Do I need tcp_keep alives?
Why does my connection breaks after a certain time and is no more recoverable?
My App-properties:
#
# [ Database Configuration Section ]
#
spring.jpa.database=POSTGRESQL
spring.jpa.show-sql=false
hibernate.format_sql=true
hibernate.hbm2ddl.auto=validate
spring.jpa.hibernate.naming-strategy=org.hibernate.cfg.DefaultNamingStrategy
spring.datasource.platform=postgres
spring.database.driverClassName=org.postgresql.Driver
spring.datasource.url=jdbc:postgresql://*****:5434/******
spring.datasource.username=*****
spring.datasource.password=*****
logging.file=*******.log
logging.level.=WARNING
2015-09-29 11:58:50.598 INFO 10498 --- [nio-9092-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring FrameworkServlet 'dispatcherServlet'
2015-09-29 11:58:50.598 INFO 10498 --- [nio-9092-exec-1] o.s.web.servlet.DispatcherServlet : FrameworkServlet 'dispatcherServlet': initialization started
2015-09-29 11:58:50.674 INFO 10498 --- [nio-9092-exec-1] o.s.web.servlet.DispatcherServlet : FrameworkServlet 'dispatcherServlet': initialization completed in 76 ms
2015-09-29 19:23:03.777 WARN 10498 --- [ool-3-thread-16] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: 08006
2015-09-29 19:23:03.779 ERROR 10498 --- [ool-3-thread-16] o.h.engine.jdbc.spi.SqlExceptionHelper : An I/O error occurred while sending to the backend.
2015-09-29 19:23:03.785 INFO 10498 --- [ool-3-thread-16] o.h.e.j.b.internal.AbstractBatchImpl : HHH000010: On release of batch it still contained JDBC statements
2015-09-29 19:23:03.836 ERROR 10498 --- [ool-3-thread-16] o.s.orm.jpa.JpaTransactionManager : Commit exception overridden by rollback exception
java.net.SocketException: Connection timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_25]
at java.net.SocketInputStream.read(SocketInputStream.java:150) ~[na:1.8.0_25]
at java.net.SocketInputStream.read(SocketInputStream.java:121) ~[na:1.8.0_25]
at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143) ~[postgresql-9.4-1202-jdbc41.jar!/:9.4]
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112) ~[postgresql-9.4-1202-jdbc41.jar!/:9.4]
at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:71) ~[postgresql-9.4-1202-jdbc41.jar!/:9.4]
at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:282) ~[postgresql-9.4-1202-jdbc41.jar!/:9.4]
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1718) ~[postgresql-9.4-1202-jdbc41.jar!/:9.4]
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:173) ~[postgresql-9.4-1202-jdbc41.jar!/:9.4]
... 61 common frames omitted
Wrapped by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:201) ~[postgresql-9.4-1202-jdbc41.jar!/:9.4]
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:615) ~[postgresql-9.4-1202-jdbc41.jar!/:9.4]
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:465) ~[postgresql-9.4-1202-jdbc41.jar!/:9.4]
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:411) ~[postgresql-9.4-1202-jdbc41.jar!/:9.4]
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:208) ~[hibernate-core-4.3.11.Final.jar!/:4.3.11.Final]
... 57 common frames omitted
Wrapped by: org.hibernate.exception.JDBCConnectionException: could not execute statement
EDIT:
I've added this and testing it currently:
spring.datasource.testOnBorrow=true
spring.datasource.validationQuery=SELECT 1
And removed autoReconnect.

As mentioned by #Yuki Yoshida the answer was that simple.
I added
spring.datasource.validation-query= select 1
spring.datasource.test-on-borrow=true
to my configuration and it works.
Test-on-borrow:
testOnBorrow: Default: true
The indication of whether objects will be validated
before being borrowed from the pool. If the object fails to validate,
it will be dropped from the pool, and we will attempt to borrow
another.
And
validationQuery: Default: for most db languages select 1 else see here
validationQuery
The SQL query that will be used to validate
connections from this pool before returning them to the caller. If
specified, this query MUST be an SQL SELECT statement that returns at
least one row. If not specified, connections will be validation by
calling the isValid() method.
I actually tried this already but I did not explicitly set test-on-borrow to true because of its default. Looks like you have to set it explicitly.