I am getting connection is not available error msg with below config xml mapping.Please suggest me what i am doing wrong in beolw code or this is some another issue. As per my investigation this is an intregation issue with HikariCP, Hibernate and Spring Batch.
2.0.3
5.1.6
4.0.0.RELEASE
3.0.0.RELEASE
3.0.0.RELEASE
0.5
2.3.2
4.3.5.Final
Below Configuration
#Bean
public DataSource dataSource() {
final HikariConfig config = new HikariConfig();
config.setMaximumPoolSize(1);
System.out.println("Pool size is "+config.getMaximumPoolSize());
config.setDriverClassName(environment.getRequiredProperty(PROPERTY_NAME_DATABASE_DRIVER));
config.setJdbcUrl(environment.getRequiredProperty(PROPERTY_NAME_DATABASE_URL));
config.setUsername(environment.getRequiredProperty(PROPERTY_NAME_DATABASE_USERNAME));
config.setPassword(environment.getRequiredProperty(PROPERTY_NAME_DATABASE_PASSWORD));
final HikariDataSource dataSource = new HikariDataSource(config);
return dataSource;
}
Error:--
DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Timeout failure stats (total=1, active=1, idle=0, waiting=0)
2017-01-03T11:10:30.33+0000 [App/0] OUT PDC::fre-staging/0: 2017-01-03 11:10:30.335 [task-scheduler-6] WARN o.h.e.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: null
2017-01-03T11:10:30.33+0000 [App/0] OUT PDC::fre-staging/0: 2017-01-03 11:10:30.335 [task-scheduler-6] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - HikariPool-1 - Connection is not available, request timed out after 30000ms.
2017-01-03T11:10:30.34+0000 [App/0] OUT In finally If
2017-01-03T11:10:43.89+0000 [App/0] OUT PDC::fre-staging/0: 2017-01-03 11:10:43.896 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Pool stats (total=1, active=1, idle=0, waiting=1)
2017-01-03T11:11:00.34+0000 [App/0] OUT PDC::fre-staging/0: 2017-01-03 11:11:00.340 [task-scheduler-6] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Timeout failure stats (total=1, active=1, idle=0, waiting=0)
2017-01-03T11:11:00.34+0000 [App/0] OUT PDC::fre-staging/0: 2017-01-03 11:11:00.341 [task-scheduler-6] WARN o.h.e.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: null
2017-01-03T11:11:00.34+0000 [App/0] OUT PDC::fre-staging/0: 2017-01-03 11:11:00.341 [task-scheduler-6] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - HikariPool-1 - Connection is not available, request timed out after 30000ms.
2017-01-03T11:11:00.35+0000 [App/0] OUT PDC::fre-staging/0: 2017-01-03 11:11:00.354 [task-scheduler-6] ERROR o.s.batch.core.step.AbstractStep - Encountered an error executing step emisTopupStep in job emisTopupJob
2017-01-03T11:11:00.35+0000 [App/0] OUT javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Could not open connection.
Please help me out.
You are setting the pool max size to 1 with the line
config.setMaximumPoolSize(1);
My guess is that is your problem. Try increasing it, just don't over do it. The defaults is 10 which is good for most cases.
Related
I have a spring boot application using hikari and a Mysql database.
There is my hikari configuration :
HikariPool-1 - configuration:
allowPoolSuspension.............false
autoCommit......................true
catalog.........................none
connectionInitSql...............none
connectionTestQuery.............none
connectionTimeout...............300000
dataSource......................none
dataSourceClassName.............none
dataSourceJNDI..................none
dataSourceProperties............{password=<masked>}
driverClassName................."org.mariadb.jdbc.Driver"
exceptionOverrideClassName......none
healthCheckProperties...........{}
healthCheckRegistry.............none
idleTimeout.....................600000
initializationFailTimeout.......1
isolateInternalQueries..........false
jdbcUrl.........................jdbc:mysql://mydatabase?rewriteBatchedStatements=true
leakDetectionThreshold..........0
maxLifetime.....................27000000
maximumPoolSize.................70
metricRegistry..................none
metricsTrackerFactory...........none
minimumIdle.....................20
password........................<masked>
poolName........................"HikariPool-1"
readOnly........................false
registerMbeans..................false
scheduledExecutor...............none
schema..........................none
threadFactory...................internal
transactionIsolation............default
username........................"user_name"
validationTimeout...............10000
There is my database configuration :
wait_timeout 28800
interactive_timeout 28800
innodb_lock_wait_timeout 180
connect_timeout 10
I keep getting this error and I don't understand why :
2022-10-20 14:39:37,465 WARN c8c20a5b-7877-4830-bac8-ece994ae84e5 c.z.h.p.ProxyConnection: HikariPool-1 - Connection org.mariadb.jdbc.MariaDbConnection#5737d116 marked as broken because of SQLSTATE(08000), ErrorCode(0)
java.sql.SQLNonTransientConnectionException: (conn=10076858) Connexion terminée par expiration du délai d'attente (Read failed)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:153)
at org.mariadb.jdbc.MariaDbStatement.executeExceptionEpilogue(MariaDbStatement.java:274)
at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:363)
at org.mariadb.jdbc.MariaDbStatement.execute(MariaDbStatement.java:501)
at org.mariadb.jdbc.MariaDbConnection.commit(MariaDbConnection.java:721)
at com.zaxxer.hikari.pool.ProxyConnection.commit(ProxyConnection.java:387)
at com.zaxxer.hikari.pool.HikariProxyConnection.commit(HikariProxyConnection.java)
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.commit(AbstractLogicalConnectionImplementor.java:86)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:282)
at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:101)
at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:562)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:743)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:711)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:654)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:407)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692)
The connection 5737d116 referenced in the stacktrace has been created just 2 hours before :
2022-10-20 12:37:30,333 DEBUG c.z.h.p.HikariPool: HikariPool-1 - Added connection org.mariadb.jdbc.MariaDbConnection#5737d116
And here is the state of the pool of connection just before the exception occured :
2022-10-20 14:39:00,600 DEBUG c.z.h.p.HikariPool: HikariPool-1 - Before cleanup stats (total=21, active=1, idle=20, waiting=0)
2022-10-20 14:39:00,601 DEBUG c.z.h.p.HikariPool: HikariPool-1 - After cleanup stats (total=21, active=1, idle=20, waiting=0)
2022-10-20 14:39:00,601 DEBUG c.z.h.p.HikariPool: HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2022-10-20 14:39:02,197 DEBUG c.z.h.p.HikariPool: HikariPool-2 - Before cleanup stats (total=20, active=0, idle=20, waiting=0)
2022-10-20 14:39:02,197 DEBUG c.z.h.p.HikariPool: HikariPool-2 - After cleanup stats (total=20, active=0, idle=20, waiting=0)
2022-10-20 14:39:02,197 DEBUG c.z.h.p.HikariPool: HikariPool-2 - Fill pool skipped, pool is at sufficient level.
2022-10-20 14:39:30,601 DEBUG c.z.h.p.HikariPool: HikariPool-1 - Before cleanup stats (total=21, active=1, idle=20, waiting=0)
2022-10-20 14:39:30,601 DEBUG c.z.h.p.HikariPool: HikariPool-1 - After cleanup stats (total=21, active=1, idle=20, waiting=0)
2022-10-20 14:39:30,601 DEBUG c.z.h.p.HikariPool: HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2022-10-20 14:39:32,198 DEBUG c.z.h.p.HikariPool: HikariPool-2 - Before cleanup stats (total=20, active=0, idle=20, waiting=0)
2022-10-20 14:39:32,198 DEBUG c.z.h.p.HikariPool: HikariPool-2 - After cleanup stats (total=20, active=0, idle=20, waiting=0)
Thanks in advance for your help
After trying to figure out whats wrong with our app, I have decided to look here for some help.
We have a Java app which uses below packages for HikariCP database pool
+--- com.custom.gwtext:gwtext-server
| +--- org.hibernate:hibernate-hikaricp:5.4.6.Final
| | +--- org.hibernate:hibernate-core:5.4.6.Final (*)
| | \--- com.zaxxer:HikariCP:3.2.0 -> 3.3.0
| | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.36
| +--- com.zaxxer:HikariCP:3.3.0 (*)
The issue is that we are currently trying to migrate to Kubernetes and we started to see a lot of java.sql.SQLException: Connection is closed errors after application is running for several hours. After enabling Hikari DEBUG log and adding leakDetectionThreshold we started to notice a few things.
This is the configuration for HikariCP
00:06:05.521 [main] DEBUG com.zaxxer.hikari.HikariConfig - HikariPool-2 - configuration:
00:06:05.522 [main] DEBUG com.zaxxer.hikari.HikariConfig - allowPoolSuspension.............false
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - autoCommit......................false
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - catalog.........................none
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - connectionInitSql...............none
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - connectionTestQuery.............none
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - connectionTimeout...............30000
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSource......................none
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceClassName.............none
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceJNDI..................none
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceProperties............{password=<masked>}
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - driverClassName................."oracle.jdbc.OracleDriver"
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - healthCheckProperties...........{}
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - healthCheckRegistry.............none
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - idleTimeout.....................600000
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - initializationFailTimeout.......1
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - isolateInternalQueries..........false
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - jdbcUrl.........................jdbc:oracle:thin:#1.1.1.1:1521:test
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - leakDetectionThreshold..........200000
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - maxLifetime.....................18000000
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - maximumPoolSize.................90
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - metricRegistry..................none
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - metricsTrackerFactory...........none
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - minimumIdle.....................15
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - password........................<masked>
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - poolName........................"HikariPool-2"
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - readOnly........................false
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - registerMbeans..................false
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - scheduledExecutor...............none
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - schema..........................none
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - threadFactory...................internal
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - transactionIsolation............default
00:06:05.528 [main] DEBUG com.zaxxer.hikari.HikariConfig - username........................"user"
00:06:05.528 [main] DEBUG com.zaxxer.hikari.HikariConfig - validationTimeout...............5000
After starting up the Java app and logging in with the user, after some minutes (probably the leakDetectionThreshold) we notice below log entry, which says that there is 2 leaks right away
[HikariPool-2 housekeeper] WARN com.zaxxer.hikari.pool.ProxyLeakTask - Connection leak detection triggered for oracle.jdbc.driver.T4CConnection#5ae99303 on thread pool-10-thread-1, stack trace follows
java.lang.Exception: Apparent connection leak detected
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:104)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:134)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:250)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:258)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246)
at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83)
at org.hibernate.internal.AbstractSharedSessionContract.beginTransaction(AbstractSharedSessionContract.java:467)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:350)
at com.sun.proxy.$Proxy326.beginTransaction(Unknown Source)
at com.custom.gwtext.server.utils.HibernateUtilImpl.startTransaction(HibernateUtilImpl.java:335)
at com.custom.gwtext.server.utils.HibernateUtilImpl.startTransactionIfNeeded(HibernateUtilImpl.java:330)
at com.custom.gwtext.server.utils.HibernateUtilImpl.getSession(HibernateUtilImpl.java:70)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.reserveIdBatch(DatabaseSequenceIdPoolDataUtil.java:29)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.reserveIdsInDatabase(DatabaseSequenceIdPool.java:106)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.access$300(DatabaseSequenceIdPool.java:24)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool$ReserveIdsTask.run(DatabaseSequenceIdPool.java:125)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
and a few minutes later second one
[HikariPool-2 housekeeper] WARN com.zaxxer.hikari.pool.ProxyLeakTask - Connection leak detection triggered for oracle.jdbc.driver.T4CConnection#7b911a8f on thread pool-6-thread-1, stack trace follows
java.lang.Exception: Apparent connection leak detected
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:104)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:134)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:250)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:258)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246)
at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83)
at org.hibernate.internal.AbstractSharedSessionContract.beginTransaction(AbstractSharedSessionContract.java:467)
at jdk.internal.reflect.GeneratedMethodAccessor166.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:350)
at com.sun.proxy.$Proxy326.beginTransaction(Unknown Source)
at com.custom.gwtext.server.utils.HibernateUtilImpl.startTransaction(HibernateUtilImpl.java:335)
at com.custom.gwtext.server.utils.HibernateUtilImpl.startTransactionIfNeeded(HibernateUtilImpl.java:330)
at com.custom.gwtext.server.utils.HibernateUtilImpl.getSession(HibernateUtilImpl.java:70)
at com.custom.gwtext.server.utils.DbUtilImpl.doWork(DbUtilImpl.java:516)
at com.custom.gwtext.server.utils.DbUtilOracleImpl.executeQueryDBSpecific(DbUtilOracleImpl.java:117)
at com.custom.gwtext.server.utils.DbUtilOracleImpl.executeQueryDBSpecific(DbUtilOracleImpl.java:67)
at com.custom.gwtext.server.utils.DbUtilImpl.executeQueryInternal(DbUtilImpl.java:266)
at com.custom.gwtext.server.utils.DbUtilImpl.executeQuery(DbUtilImpl.java:173)
at com.custom.gwtext.server.query.QueriesDefinitionRegistryImpl.getData(QueriesDefinitionRegistryImpl.java:147)
at com.custom.gwtext.server.query.QueriesDefinitionRegistryImpl.getData(QueriesDefinitionRegistryImpl.java:74)
at com.custom.pins.server.data.helper.QueryHelper.executeSelectFirstRow(QueryHelper.java:54)
at com.custom.pins.server.data.helper.QueryHelper.selectFirstRow(QueryHelper.java:36)
at com.custom.pins.server.data.helper.QueryHelper.selectFirstValueRequired(QueryHelper.java:80)
at com.custom.pins.server.business.bdt.BusinessDataTableDataUtil.existsBusinessTable(BusinessDataTableDataUtil.java:20)
at com.custom.pins.server.business.bdt.BusinessDataTableBuilderImpl.existsTable(BusinessDataTableBuilderImpl.java:225)
at com.custom.pins.server.security.audit.ws.WebServiceLoggerConfigImpl.createWsRequestConfig(WebServiceLoggerConfigImpl.java:116)
at com.custom.pins.server.security.audit.ws.WebServiceLoggerConfigImpl.getWsRequestConfig(WebServiceLoggerConfigImpl.java:75)
at com.custom.pins.server.security.audit.ws.WebServiceAuditLogService.getWsRequestSpecificConfiguration(WebServiceAuditLogService.java:88)
at com.custom.pins.server.security.audit.ws.WebServiceAuditLogService.logIfConfigEnabled(WebServiceAuditLogService.java:71)
at com.custom.pins.server.security.audit.ws.WebServiceAuditLogService.lambda$getInternalWsLogConsumer$0(WebServiceAuditLogService.java:62)
at com.custom.gwtext.server.security.audit.AuditLogServiceImpl.lambda$createEventHandler$2(AuditLogServiceImpl.java:90)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:150)
at java.base/java.lang.Thread.run(Thread.java:834)
after which they both stay active for a long time
[HikariPool-2 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-2 - After cleanup stats (total=17, active=2, idle=15, waiting=0)
Then after a few hours, this can be 3-10 hours, so no specific range, maybe depends on activity etc, no idea, this pops up in logs
[pool-10-thread-1] WARN c.zaxxer.hikari.pool.ProxyConnection - HikariPool-2 - Connection oracle.jdbc.driver.T4CConnection#5ae99303 marked as broken because of SQLSTATE(08006), ErrorCode(17002)
java.sql.SQLRecoverableException: IO Error: Connection reset by peer
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:929)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1119)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3780)
at oracle.jdbc.driver.T4CCallableStatement.executeInternal(T4CCallableStatement.java:1300)
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3887)
at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4230)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1079)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
at com.zaxxer.hikari.pool.HikariProxyCallableStatement.execute(HikariProxyCallableStatement.java)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.lambda$reserveIdBatch$0(DatabaseSequenceIdPoolDataUtil.java:34)
at org.hibernate.jdbc.WorkExecutor.executeWork(WorkExecutor.java:37)
at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2306)
at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2303)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:306)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2325)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2310)
at jdk.internal.reflect.GeneratedMethodAccessor128.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:350)
at com.sun.proxy.$Proxy326.doWork(Unknown Source)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.reserveIdBatch(DatabaseSequenceIdPoolDataUtil.java:29)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.reserveIdsInDatabase(DatabaseSequenceIdPool.java:106)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.access$300(DatabaseSequenceIdPool.java:24)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool$ReserveIdsTask.run(DatabaseSequenceIdPool.java:125)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Connection reset by peer
at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:245)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:355)
at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:144)
at oracle.net.ns.NIOHeader.readHeaderBuffer(NIOHeader.java:82)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:139)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:101)
at oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:80)
at oracle.jdbc.driver.T4CMAREngineNIO.prepareForReading(T4CMAREngineNIO.java:98)
at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:534)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:485)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612)
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:223)
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:56)
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:907)
... 29 common frames omitted
[pool-10-thread-1] INFO com.zaxxer.hikari.pool.ProxyLeakTask - Previously reported leaked connection oracle.jdbc.driver.T4CConnection#5ae99303 on thread pool-10-thread-1 was returned to the pool (unleaked)
[HikariPool-2 connection closer] DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-2 - Closing connection oracle.jdbc.driver.T4CConnection#5ae99303: (connection is broken)
Also, it's not active anymore (the active one is the second leaked one T4CConnection#7b911a8f on thread pool-6-thread-1 which somehow does not reset).
And right after this happens, our Java app starts to throw a lot of (even like 10 per second, sometimes less) below errors and becomes very slow
[pool-10-thread-1] WARN o.h.e.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: null
[pool-10-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - Connection is closed
[pool-10-thread-1] ERROR l.p.p.s.i.DatabaseSequenceIdPool - error executing work
org.hibernate.exception.GenericJDBCException: error executing work
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:47)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:311)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2325)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2310)
at jdk.internal.reflect.GeneratedMethodAccessor128.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:350)
at com.sun.proxy.$Proxy326.doWork(Unknown Source)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.reserveIdBatch(DatabaseSequenceIdPoolDataUtil.java:29)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.reserveIdsInDatabase(DatabaseSequenceIdPool.java:106)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.access$300(DatabaseSequenceIdPool.java:24)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool$ReserveIdsTask.run(DatabaseSequenceIdPool.java:125)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.sql.SQLException: Connection is closed
at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:489)
at com.sun.proxy.$Proxy307.prepareCall(Unknown Source)
at com.zaxxer.hikari.pool.ProxyConnection.prepareCall(ProxyConnection.java:290)
at com.zaxxer.hikari.pool.HikariProxyConnection.prepareCall(HikariProxyConnection.java)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.lambda$reserveIdBatch$0(DatabaseSequenceIdPoolDataUtil.java:31)
at org.hibernate.jdbc.WorkExecutor.executeWork(WorkExecutor.java:37)
at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2306)
at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2303)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:306)
... 16 common frames omitted
So, it seems like although T4CConnection#5ae99303 is marked as broken, it's never evicted and the app still tries to use it.
Can this be related to HikariCP? Looks like we are using quite outdated version of Hikari and Hibernate, but somehow those versions are baked in some custom com.custom package and not sure how we can "overwrite" those versions. We are using Gradle for building the WAR file.
Oh and also just to add, there are no such issues when running without Kubernetes. Maybe this is related to network/firewalls, but somehow I think HikariCP should work around those broken connections in case they happen.
I would really appreciate the help or even some pointers where/what to look for.
i have following problem:
I am currently writing a spring boot application that provides a rest endpoint and also constantly evaluates some data stored inside a remote MySql database that i have to reach through a SSH tunnel. I use Jcraft library to create that tunnel and dynamically assign a free port that my data source is using to connect to the database through the tunnel. The port cannot be static.
The data source itself is a Hikari connection pool.
Every once in a while though something happens and i lose my connection to the database.
I want to create a failsafe that basicly catches any exceptions related to any issues with the database communication, automatically shuts down the old tunnel + data source and then creates a new tunnel + recreates the data source.
My current approach was as following:
Basicly just recreate the the tunnel is it does not exist anymore and then close the connection pool + restarting it again. After that i set the new data source inside every JdbcTemplate thats autowired inside the DAO's.
My data source configuration bean:
import com.jcraft.jsch.JSchException;
import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import net.candidatis.tiertwo.data.*;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.util.concurrent.TimeUnit;
#Slf4j
#Configuration
#Data
public class ConfigDataSource {
HikariDataSource ds;
private SshSession sshSession;
private DaoCustom daoCustom;
private DaoPortals daoPortals;
private DaoBlacklist daoBlacklist;
private DaoDbChecker daoDbChecker;
private DaoPublishing daoPublishing;
private DaoTempJobs daoTempJobs;
public ConfigDataSource(SshSession sshSession){
this.sshSession = sshSession;
}
#Bean
public HikariDataSource dataSource(){
ds = new HikariDataSource(getConfig());
return ds;
}
public void reloadDataSource(){
log.info("Reloading the Connection pool.");
if(ds != null && !ds.isClosed()){
ds.close();
}
ds = new HikariDataSource(getConfig());
this.daoCustom.getJdbcTemplate().setDataSource(ds);
this.daoPortals.getJdbcTemplate().setDataSource(ds);
this.daoBlacklist.getJdbcTemplate().setDataSource(ds);;
this.daoDbChecker.getJdbcTemplate().setDataSource(ds);
this.daoPublishing.getJdbcTemplate().setDataSource(ds);
this.daoPublishing.getNamedJdbcTemplate().getJdbcTemplate().setDataSource(ds);
this.daoTempJobs.getJdbcTemplate().setDataSource(ds);
}
private HikariConfig getConfig(){
try {
this.sshSession.createSessionSSH();
} catch (JSchException e) {
e.printStackTrace();
log.error("Could not create SSH session");
}
HikariConfig config = new HikariConfig();
config.setJdbcUrl( Constants.DB_JDBC_URL + this.sshSession.getAssignedPort());
config.setUsername( Constants.DB_USER );
config.setPassword( Constants.DB_PASSWORD );
config.setMaximumPoolSize(2);
config.setMaxLifetime(TimeUnit.MINUTES.toMillis(5)); // default wait_timeout of MySQL = 28800 seconds = 8 hours -> must be less
config.setConnectionTimeout(TimeUnit.MINUTES.toMillis(10)); // Maximum number of milliseconds that a client will wait for a connection from the pool. Throws SQLException
config.setValidationTimeout(TimeUnit.MINUTES.toMillis(8)); // This property controls the maximum amount of time that a connection will be tested for aliveness.
config.setLeakDetectionThreshold(30000);
return config;
}
}
The exception handling if anything should happen is basicly just the reloadDataSource method.
It creates a new pool using the configuration with the new assigned port and updates all data sources of all JdbcTemplate classes..
And it ALMOST works. I can reload the pool and i can then use SELECT statements on the database to fetch data, however i dont seem to be able to INSERT,UPDATE or DELETE data.
Here a part of my hikariCP log at startup(not reloading):
2021-03-31 13:18:05.051 DEBUG 471740 --- [ restartedMain] com.zaxxer.hikari.util.DriverDataSource : Loaded driver with class name com.mysql.cj.jdbc.Driver for jdbcUrl=jdbc:mysql://localhost:41749
2021-03-31 13:18:05.566 DEBUG 471740 --- [ restartedMain] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.mysql.cj.jdbc.ConnectionImpl#647dd875
2021-03-31 13:18:05.566 INFO 471740 --- [ restartedMain] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2021-03-31 13:18:05.666 DEBUG 471740 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=1, active=1, idle=0, waiting=0)
2021-03-31 13:18:05.714 DEBUG 471740 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.mysql.cj.jdbc.ConnectionImpl#1c9d2fe4
2021-03-31 13:18:05.714 DEBUG 471740 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - After adding stats (total=2, active=1, idle=1, waiting=0)
..and here again after the reload:
2021-03-31 13:18:24.808 INFO 471740 --- [ restartedMain] n.c.tiertwo.config.ConfigDataSource : Reloading the Connection pool.
2021-03-31 13:18:24.808 INFO 471740 --- [ restartedMain] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2021-03-31 13:18:24.809 DEBUG 471740 --- [ restartedMain] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Before shutdown stats (total=2, active=0, idle=2, waiting=0)
2021-03-31 13:18:24.809 DEBUG 471740 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl#647dd875: (connection evicted)
2021-03-31 13:18:24.812 DEBUG 471740 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl#1c9d2fe4: (connection evicted)
2021-03-31 13:18:24.812 DEBUG 471740 --- [ restartedMain] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - After shutdown stats (total=0, active=0, idle=0, waiting=0)
2021-03-31 13:18:24.812 INFO 471740 --- [ restartedMain] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
2021-03-31 13:18:34.946 INFO 471740 --- [ restartedMain] com.zaxxer.hikari.HikariDataSource : HikariPool-2 - Starting...
2021-03-31 13:18:34.946 DEBUG 471740 --- [ restartedMain] com.zaxxer.hikari.util.DriverDataSource : Loaded driver with class name com.mysql.cj.jdbc.Driver for jdbcUrl=jdbc:mysql://localhost:35177
2021-03-31 13:18:35.691 DEBUG 471740 --- [ restartedMain] com.zaxxer.hikari.pool.HikariPool : HikariPool-2 - Added connection com.mysql.cj.jdbc.ConnectionImpl#153f9ff2
2021-03-31 13:18:35.692 INFO 471740 --- [ restartedMain] com.zaxxer.hikari.HikariDataSource : HikariPool-2 - Start completed.
2021-03-31 13:18:35.710 INFO 471740 --- [ restartedMain] n.c.tiertwo.controllers.MainController : Fetching TempJob Block from database
2021-03-31 13:18:35.792 DEBUG 471740 --- [l-2 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-2 - Pool stats (total=1, active=1, idle=0, waiting=0)
2021-03-31 13:18:35.978 DEBUG 471740 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-2 - Added connection com.mysql.cj.jdbc.ConnectionImpl#22eb1758
2021-03-31 13:18:35.978 DEBUG 471740 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-2 - After adding stats (total=2, active=1, idle=1, waiting=0)
Again: I can use SELECT statements so the pool should actually exist. However on INSERT,DELTE or any other queries i get following errors:
at org.springframework.jdbc.core.metadata.TableMetaDataProviderFactory.createMetaDataProvider(TableMetaDataProviderFactory.java:86)
at org.springframework.jdbc.core.metadata.TableMetaDataContext.processMetaData(TableMetaDataContext.java:171)
at org.springframework.jdbc.core.simple.AbstractJdbcInsert.compileInternal(AbstractJdbcInsert.java:277)
at org.springframework.jdbc.core.simple.AbstractJdbcInsert.compile(AbstractJdbcInsert.java:261)
at org.springframework.jdbc.core.simple.AbstractJdbcInsert.checkCompiled(AbstractJdbcInsert.java:309)
at org.springframework.jdbc.core.simple.AbstractJdbcInsert.doExecuteAndReturnKey(AbstractJdbcInsert.java:368)
at org.springframework.jdbc.core.simple.SimpleJdbcInsert.executeAndReturnKey(SimpleJdbcInsert.java:127)
at net.candidatis.tiertwo.data.DaoPublishing.publish(DaoPublishing.java:220)
at net.candidatis.tiertwo.data.DaoPublishing$$FastClassBySpringCGLIB$$32c38d56.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
at net.candidatis.tiertwo.data.DaoPublishing$$EnhancerBySpringCGLIB$$bdbc3342.publish(<generated>)
at net.candidatis.tiertwo.services.Publisher.publish(Publisher.java:150)
at net.candidatis.tiertwo.services.Publisher.publishEvaluation(Publisher.java:67)
at net.candidatis.tiertwo.services.Publishing.run(Publishing.java:16)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta-data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:363)
at org.springframework.jdbc.core.metadata.TableMetaDataProviderFactory.createMetaDataProvider(TableMetaDataProviderFactory.java:52)
... 25 more
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:337)
... 26 more
Caused by: java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:96)
at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:158)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:116)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
... 27 more
org.springframework.dao.DataAccessResourceFailureException: Error retrieving database meta-data; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta-data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.
at org.springframework.jdbc.core.metadata.TableMetaDataProviderFactory.createMetaDataProvider(TableMetaDataProviderFactory.java:86)
at org.springframework.jdbc.core.metadata.TableMetaDataContext.processMetaData(TableMetaDataContext.java:171)
at org.springframework.jdbc.core.simple.AbstractJdbcInsert.compileInternal(AbstractJdbcInsert.java:277)
at org.springframework.jdbc.core.simple.AbstractJdbcInsert.compile(AbstractJdbcInsert.java:261)
at org.springframework.jdbc.core.simple.AbstractJdbcInsert.checkCompiled(AbstractJdbcInsert.java:309)
at org.springframework.jdbc.core.simple.AbstractJdbcInsert.doExecuteAndReturnKey(AbstractJdbcInsert.java:368)
at org.springframework.jdbc.core.simple.SimpleJdbcInsert.executeAndReturnKey(SimpleJdbcInsert.java:127)
at net.candidatis.tiertwo.data.DaoPublishing.publish(DaoPublishing.java:220)
at net.candidatis.tiertwo.data.DaoPublishing$$FastClassBySpringCGLIB$$32c38d56.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
at net.candidatis.tiertwo.data.DaoPublishing$$EnhancerBySpringCGLIB$$bdbc3342.publish(<generated>)
at net.candidatis.tiertwo.services.Publisher.publish(Publisher.java:150)
at net.candidatis.tiertwo.services.Publisher.publishEvaluation(Publisher.java:67)
at net.candidatis.tiertwo.services.Publishing.run(Publishing.java:16)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta-data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:363)
at org.springframework.jdbc.core.metadata.TableMetaDataProviderFactory.createMetaDataProvider(TableMetaDataProviderFactory.java:52)
... 25 more
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:337)
... 26 more
Caused by: java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:96)
at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:158)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:116)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
... 27 more
org.springframework.dao.DataAccessResourceFailureException: Error retrieving database meta-data; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta-data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.
at org.springframework.jdbc.core.metadata.TableMetaDataProviderFactory.createMetaDataProvider(TableMetaDataProviderFactory.java:86)
at org.springframework.jdbc.core.metadata.TableMetaDataContext.processMetaData(TableMetaDataContext.java:171)
at org.springframework.jdbc.core.simple.AbstractJdbcInsert.compileInternal(AbstractJdbcInsert.java:277)
at org.springframework.jdbc.core.simple.AbstractJdbcInsert.compile(AbstractJdbcInsert.java:261)
at org.springframework.jdbc.core.simple.AbstractJdbcInsert.checkCompiled(AbstractJdbcInsert.java:309)
at org.springframework.jdbc.core.simple.AbstractJdbcInsert.doExecuteAndReturnKey(AbstractJdbcInsert.java:368)
at org.springframework.jdbc.core.simple.SimpleJdbcInsert.executeAndReturnKey(SimpleJdbcInsert.java:127)
at net.candidatis.tiertwo.data.DaoPublishing.publish(DaoPublishing.java:220)
at net.candidatis.tiertwo.data.DaoPublishing$$FastClassBySpringCGLIB$$32c38d56.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
at net.candidatis.tiertwo.data.DaoPublishing$$EnhancerBySpringCGLIB$$bdbc3342.publish(<generated>)
at net.candidatis.tiertwo.services.Publisher.publish(Publisher.java:150)
at net.candidatis.tiertwo.services.Publisher.publish(Publisher.java:171)
at net.candidatis.tiertwo.services.Publisher.publishEvaluation(Publisher.java:67)
at net.candidatis.tiertwo.services.Publishing.run(Publishing.java:16)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta-data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:363)
at org.springframework.jdbc.core.metadata.TableMetaDataProviderFactory.createMetaDataProvider(TableMetaDataProviderFactory.java:52)
... 26 more
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:337)
... 27 more
Caused by: java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:96)
at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:158)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:116)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
... 28 more
org.springframework.dao.DataAccessResourceFailureException: Error retrieving database meta-data; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta-data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.
at org.springframework.jdbc.core.metadata.TableMetaDataProviderFactory.createMetaDataProvider(TableMetaDataProviderFactory.java:86)
at org.springframework.jdbc.core.metadata.TableMetaDataContext.processMetaData(TableMetaDataContext.java:171)
at org.springframework.jdbc.core.simple.AbstractJdbcInsert.compileInternal(AbstractJdbcInsert.java:277)
at org.springframework.jdbc.core.simple.AbstractJdbcInsert.compile(AbstractJdbcInsert.java:261)
at org.springframework.jdbc.core.simple.AbstractJdbcInsert.checkCompiled(AbstractJdbcInsert.java:309)
at org.springframework.jdbc.core.simple.AbstractJdbcInsert.doExecuteAndReturnKey(AbstractJdbcInsert.java:368)
at org.springframework.jdbc.core.simple.SimpleJdbcInsert.executeAndReturnKey(SimpleJdbcInsert.java:127)
at net.candidatis.tiertwo.data.DaoPublishing.publish(DaoPublishing.java:220)
at net.candidatis.tiertwo.data.DaoPublishing$$FastClassBySpringCGLIB$$32c38d56.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
at net.candidatis.tiertwo.data.DaoPublishing$$EnhancerBySpringCGLIB$$bdbc3342.publish(<generated>)
at net.candidatis.tiertwo.services.Publisher.publish(Publisher.java:150)
at net.candidatis.tiertwo.services.Publisher.publish(Publisher.java:171)
at net.candidatis.tiertwo.services.Publisher.publishEvaluation(Publisher.java:67)
at net.candidatis.tiertwo.services.Publishing.run(Publishing.java:16)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta-data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:363)
at org.springframework.jdbc.core.metadata.TableMetaDataProviderFactory.createMetaDataProvider(TableMetaDataProviderFactory.java:52)
... 26 more
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:337)
... 27 more
Caused by: java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:96)
at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:158)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:116)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
... 28 more
As you can see it is still trying to use the HikariPool #1 when it actually should use the new #2 that i have set previousely.
Any ideas why it is behaving that way ? Could it be that i cannot switch the datasource of the jdbcTemplate that is provided by the Spring container once it is initialized ? If i cannot change it why can i use a SELECT query ? Is there generally a much better way to just globally update the data source for all my DB operations ? I really need to find a solid solution for those database connection issues.
Thanks for anything.
I am thankful for any help.
See if this answer by #JonBates helps:
https://stackoverflow.com/a/64375936/5917837
My application starts in INFO: Server startup in [337,971] milliseconds
I have tried setting hibernate.temp.use_jdbc_metadata_defaults to false.
I have tried setting hibernate.dialect to org.hibernate.dialect.PostgreSQLDialect
I can connect using pgadmin to Postgres server in ms and also can query in short period of time.
The problem is in application.
jdbc.prefetchsize=10
jdbc.maximumPoolSize=1
jdbc.minimumIdle=0
jdbc.idleTimeout=30000
<property name="jpaProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.PostgreSQLDialect</prop>
<prop key="hibernate.show_sql">true</prop>
<prop key="hibernate.format_sql">false</prop>
<prop key="hibernate.generate_statistics">false</prop>
<prop key="hibernate.auditable">true</prop>
<prop key="org.hibernate.envers.audit_table_prefix">Z_</prop>
<prop key="hibernate.temp.use_jdbc_metadata_defaults">false</prop>
</props>
</property>
Here is the console log for the problematic part:
16:41:07.395 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Before cleanup stats (total=1, active=0, idle=1, waiting=0)
16:41:07.396 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - After cleanup stats (total=1, active=0, idle=1, waiting=0)
16:41:37.401 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Before cleanup stats (total=1, active=0, idle=1, waiting=0)
16:41:37.401 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - After cleanup stats (total=0, active=0, idle=0, waiting=0)
16:41:37.401 [HikariPool-1 connection closer] DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Closing connection org.postgresql.jdbc.PgConnection#4bfc28ec: (connection has passed idleTimeout)
16:42:07.401 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Before cleanup stats (total=0, active=0, idle=0, waiting=0)
16:42:07.402 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - After cleanup stats (total=0, active=0, idle=0, waiting=0)
16:42:37.403 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Before cleanup stats (total=0, active=0, idle=0, waiting=0)
16:42:37.403 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - After cleanup stats (total=0, active=0, idle=0, waiting=0)
16:43:07.404 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Before cleanup stats (total=0, active=0, idle=0, waiting=0)
16:43:07.404 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - After cleanup stats (total=0, active=0, idle=0, waiting=0)
16:43:37.407 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Before cleanup stats (total=0, active=0, idle=0, waiting=0)
16:43:37.408 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - After cleanup stats (total=0, active=0, idle=0, waiting=0)
16:44:07.408 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Before cleanup stats (total=0, active=0, idle=0, waiting=0)
16:44:07.409 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - After cleanup stats (total=0, active=0, idle=0, waiting=0)
Jul 16, 2019 4:44:29 PM com.etstur.cs.backend.datasrvds.impl.RuleContainer getMap
INFO: Rule map refilled in 219127ms
16:44:30.078 [HikariPool-1 connection adder] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection#110cb9bc
Hibernate: select autocomple0_.id as id1_13_, autocomple0_.status as status2_13_, autocomple0_.text as text3_13_ from AutoCompleteWhiteList autocomple0_ where autocomple0_.status='ACTIVE'
Hibernate: select hotel_id from black_friday_hotels
Initializing ExecutorService 'taskExecutor'
Root WebApplicationContext initialized in 324837 ms
i have threaded system. too many (thousands)
Sometimes we have had a database bottleneck.
For analyse this bottleneck i had opened log on debug mode. And i saw this :
Hikari loaded...
springHikariCP - configuration:
allowPoolSuspension.............false
autoCommit......................true
catalog.........................none
connectionInitSql...............none
connectionTestQuery............."SELECT 1"
connectionTimeout...............30000
dataSource......................none
dataSourceClassName.............none
dataSourceJNDI..................none
dataSourceProperties............{user=blabla, url=jdbc:jtds:sqlserver:
driverClassName................."net.sourceforge.jtds.jdbc.Driver"
healthCheckProperties...........{}
healthCheckRegistry.............none
idleTimeout.....................600000
initializationFailFast..........true
initializationFailTimeout.......1
isolateInternalQueries..........false
jdbc4ConnectionTest.............false
jdbcUrl........................."jdbc:jtds:sqlserver://blabla"
leakDetectionThreshold..........0
maxLifetime.....................1800000
maximumPoolSize.................3000
metricRegistry..................none
metricsTrackerFactory...........none
minimumIdle.....................0
password........................<masked>
poolName........................"springHikariCP"
readOnly........................false
registerMbeans..................false
scheduledExecutor...............none
scheduledExecutorService........internal
threadFactory...................internal
transactionIsolation............default
username........................none
validationTimeout...............5000
springHikariCP - Starting...
and there is hundered 0 statistics until end of the log...
Pool stats (total=0, active=0, idle=0, waiting=0)
Pool stats (total=0, active=0, idle=0, waiting=0)
Pool stats (total=0, active=0, idle=0, waiting=0)
.....
How is possible it? There is too many db operations update,delete,select,insert executing. What means Total=0 connections?
Hadn't we set-load Hikari configuration properly?