I am facing this problem for 2 hours.
The problem is I have other classes that are working well. I don't know why this error is happening for the current class.
org.hibernate.exception.JDBCConnectionException: unable to obtain isolated JDBC connection
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 351ms.
But the code works for other tables(other classes).
It was working so fine until today. Is there anyone that faced this problem before?
2020-03-13 13:52:25,392 [main] WARN com.zaxxer.hikari.HikariConfig -ScraperPool - idleTimeout is less than 10000ms, setting to default 600000ms.
2020-03-13 13:52:25,400 [main] DEBUG com.zaxxer.hikari.HikariConfig -ScraperPool - configuration:
2020-03-13 13:52:25,406 [main] DEBUG com.zaxxer.hikari.HikariConfig -allowPoolSuspension.............false
2020-03-13 13:52:25,407 [main] DEBUG com.zaxxer.hikari.HikariConfig -autoCommit......................true
2020-03-13 13:52:25,407 [main] DEBUG com.zaxxer.hikari.HikariConfig -catalog.........................none
2020-03-13 13:52:25,408 [main] DEBUG com.zaxxer.hikari.HikariConfig -connectionInitSql...............none
2020-03-13 13:52:25,408 [main] DEBUG com.zaxxer.hikari.HikariConfig -connectionTestQuery.............none
2020-03-13 13:52:25,409 [main] DEBUG com.zaxxer.hikari.HikariConfig -connectionTimeout...............10000
2020-03-13 13:52:25,409 [main] DEBUG com.zaxxer.hikari.HikariConfig -dataSource......................none
2020-03-13 13:52:25,409 [main] DEBUG com.zaxxer.hikari.HikariConfig -dataSourceClassName.............none
2020-03-13 13:52:25,410 [main] DEBUG com.zaxxer.hikari.HikariConfig -dataSourceJNDI..................none
2020-03-13 13:52:25,411 [main] DEBUG com.zaxxer.hikari.HikariConfig -dataSourceProperties............{password=<masked>, prepStmtCacheSqlLimit=2048, cachePrepStmts=true, prepStmtCacheSize=250, leakDetectionThreshold=15000}
2020-03-13 13:52:25,412 [main] DEBUG com.zaxxer.hikari.HikariConfig -driverClassName.................none
2020-03-13 13:52:25,412 [main] DEBUG com.zaxxer.hikari.HikariConfig -healthCheckProperties...........{}
2020-03-13 13:52:25,412 [main] DEBUG com.zaxxer.hikari.HikariConfig -healthCheckRegistry.............none
2020-03-13 13:52:25,413 [main] DEBUG com.zaxxer.hikari.HikariConfig -idleTimeout.....................600000
2020-03-13 13:52:25,413 [main] DEBUG com.zaxxer.hikari.HikariConfig -initializationFailTimeout.......1
2020-03-13 13:52:25,413 [main] DEBUG com.zaxxer.hikari.HikariConfig -isolateInternalQueries..........false
2020-03-13 13:52:25,414 [main] DEBUG com.zaxxer.hikari.HikariConfig -jdbcUrl.........................jdbc:mysql://**********?verifyServerCertificate=false&useSSL=true&serverTimezone=UTC
2020-03-13 13:52:25,414 [main] DEBUG com.zaxxer.hikari.HikariConfig -leakDetectionThreshold..........10000
2020-03-13 13:52:25,414 [main] DEBUG com.zaxxer.hikari.HikariConfig -maxLifetime.....................1800000
2020-03-13 13:52:25,414 [main] DEBUG com.zaxxer.hikari.HikariConfig -maximumPoolSize.................100
2020-03-13 13:52:25,415 [main] DEBUG com.zaxxer.hikari.HikariConfig -metricRegistry..................none
2020-03-13 13:52:25,415 [main] DEBUG com.zaxxer.hikari.HikariConfig -metricsTrackerFactory...........none
2020-03-13 13:52:25,415 [main] DEBUG com.zaxxer.hikari.HikariConfig -minimumIdle.....................4
2020-03-13 13:52:25,415 [main] DEBUG com.zaxxer.hikari.HikariConfig -password........................<masked>
2020-03-13 13:52:25,416 [main] DEBUG com.zaxxer.hikari.HikariConfig -poolName........................"ScraperPool"
2020-03-13 13:52:25,416 [main] DEBUG com.zaxxer.hikari.HikariConfig -readOnly........................false
2020-03-13 13:52:25,416 [main] DEBUG com.zaxxer.hikari.HikariConfig -registerMbeans..................true
2020-03-13 13:52:25,417 [main] DEBUG com.zaxxer.hikari.HikariConfig -scheduledExecutor...............none
2020-03-13 13:52:25,417 [main] DEBUG com.zaxxer.hikari.HikariConfig -schema..........................none
2020-03-13 13:52:25,417 [main] DEBUG com.zaxxer.hikari.HikariConfig -threadFactory...................internal
2020-03-13 13:52:25,417 [main] DEBUG com.zaxxer.hikari.HikariConfig -transactionIsolation............default
2020-03-13 13:52:25,418 [main] DEBUG com.zaxxer.hikari.HikariConfig -username........................"xxxxxxxxxxx"
2020-03-13 13:52:25,418 [main] DEBUG com.zaxxer.hikari.HikariConfig -validationTimeout...............5000
2020-03-13 13:52:25,420 [main] INFO com.zaxxer.hikari.HikariDataSource -ScraperPool - Starting...
2020-03-13 13:52:25,452 [main] DEBUG com.zaxxer.hikari.util.DriverDataSource -Loaded driver with class name com.mysql.cj.jdbc.Driver for jdbcUrl=jdbc:mysql://*********?verifyServerCertificate=false&useSSL=true&serverTimezone=UTC
2020-03-13 13:52:27,099 [main] DEBUG com.zaxxer.hikari.pool.HikariPool -ScraperPool - Added connection com.mysql.cj.jdbc.ConnectionImpl#e84a8e1
2020-03-13 13:52:27,107 [main] INFO com.zaxxer.hikari.HikariDataSource -ScraperPool - Start completed.
2020-03-13 13:52:27,211 [ScraperPool housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool -ScraperPool - Before cleanup stats (total=1, active=0, idle=1, waiting=0)
2020-03-13 13:52:27,212 [ScraperPool housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool -ScraperPool - After cleanup stats (total=1, active=0, idle=1, waiting=0)
2020-03-13 13:52:27,280 [ScraperPool connection adder] DEBUG com.zaxxer.hikari.pool.HikariPool -ScraperPool - Added connection com.mysql.cj.jdbc.ConnectionImpl#58b0aed5
2020-03-13 13:52:27,346 [ScraperPool connection adder] DEBUG com.zaxxer.hikari.pool.HikariPool -ScraperPool - Added connection com.mysql.cj.jdbc.ConnectionImpl#285edfdd
2020-03-13 13:52:27,414 [ScraperPool connection adder] DEBUG com.zaxxer.hikari.pool.HikariPool -ScraperPool - Added connection com.mysql.cj.jdbc.ConnectionImpl#79a25ba1
2020-03-13 13:52:27,415 [ScraperPool connection adder] DEBUG com.zaxxer.hikari.pool.HikariPool -ScraperPool - After adding stats (total=4, active=0, idle=4, waiting=0)
I got this line before the error occured
DEBUG com.zaxxer.hikari.pool.HikariPool -HikariPool-1 - Timeout failure stats (total=1, active=1, idle=0, waiting=0)
Try increasing the max limit of your pool. Below code tells you that pool reached max connection limit set in the config.
DEBUG com.zaxxer.hikari.pool.HikariPool -HikariPool-1 - Timeout failure stats (total=1, active=1, idle=0, waiting=0)
Related
I have a spring boot application using hikari and a Mysql database.
There is my hikari configuration :
HikariPool-1 - configuration:
allowPoolSuspension.............false
autoCommit......................true
catalog.........................none
connectionInitSql...............none
connectionTestQuery.............none
connectionTimeout...............300000
dataSource......................none
dataSourceClassName.............none
dataSourceJNDI..................none
dataSourceProperties............{password=<masked>}
driverClassName................."org.mariadb.jdbc.Driver"
exceptionOverrideClassName......none
healthCheckProperties...........{}
healthCheckRegistry.............none
idleTimeout.....................600000
initializationFailTimeout.......1
isolateInternalQueries..........false
jdbcUrl.........................jdbc:mysql://mydatabase?rewriteBatchedStatements=true
leakDetectionThreshold..........0
maxLifetime.....................27000000
maximumPoolSize.................70
metricRegistry..................none
metricsTrackerFactory...........none
minimumIdle.....................20
password........................<masked>
poolName........................"HikariPool-1"
readOnly........................false
registerMbeans..................false
scheduledExecutor...............none
schema..........................none
threadFactory...................internal
transactionIsolation............default
username........................"user_name"
validationTimeout...............10000
There is my database configuration :
wait_timeout 28800
interactive_timeout 28800
innodb_lock_wait_timeout 180
connect_timeout 10
I keep getting this error and I don't understand why :
2022-10-20 14:39:37,465 WARN c8c20a5b-7877-4830-bac8-ece994ae84e5 c.z.h.p.ProxyConnection: HikariPool-1 - Connection org.mariadb.jdbc.MariaDbConnection#5737d116 marked as broken because of SQLSTATE(08000), ErrorCode(0)
java.sql.SQLNonTransientConnectionException: (conn=10076858) Connexion terminée par expiration du délai d'attente (Read failed)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:153)
at org.mariadb.jdbc.MariaDbStatement.executeExceptionEpilogue(MariaDbStatement.java:274)
at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:363)
at org.mariadb.jdbc.MariaDbStatement.execute(MariaDbStatement.java:501)
at org.mariadb.jdbc.MariaDbConnection.commit(MariaDbConnection.java:721)
at com.zaxxer.hikari.pool.ProxyConnection.commit(ProxyConnection.java:387)
at com.zaxxer.hikari.pool.HikariProxyConnection.commit(HikariProxyConnection.java)
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.commit(AbstractLogicalConnectionImplementor.java:86)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:282)
at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:101)
at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:562)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:743)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:711)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:654)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:407)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692)
The connection 5737d116 referenced in the stacktrace has been created just 2 hours before :
2022-10-20 12:37:30,333 DEBUG c.z.h.p.HikariPool: HikariPool-1 - Added connection org.mariadb.jdbc.MariaDbConnection#5737d116
And here is the state of the pool of connection just before the exception occured :
2022-10-20 14:39:00,600 DEBUG c.z.h.p.HikariPool: HikariPool-1 - Before cleanup stats (total=21, active=1, idle=20, waiting=0)
2022-10-20 14:39:00,601 DEBUG c.z.h.p.HikariPool: HikariPool-1 - After cleanup stats (total=21, active=1, idle=20, waiting=0)
2022-10-20 14:39:00,601 DEBUG c.z.h.p.HikariPool: HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2022-10-20 14:39:02,197 DEBUG c.z.h.p.HikariPool: HikariPool-2 - Before cleanup stats (total=20, active=0, idle=20, waiting=0)
2022-10-20 14:39:02,197 DEBUG c.z.h.p.HikariPool: HikariPool-2 - After cleanup stats (total=20, active=0, idle=20, waiting=0)
2022-10-20 14:39:02,197 DEBUG c.z.h.p.HikariPool: HikariPool-2 - Fill pool skipped, pool is at sufficient level.
2022-10-20 14:39:30,601 DEBUG c.z.h.p.HikariPool: HikariPool-1 - Before cleanup stats (total=21, active=1, idle=20, waiting=0)
2022-10-20 14:39:30,601 DEBUG c.z.h.p.HikariPool: HikariPool-1 - After cleanup stats (total=21, active=1, idle=20, waiting=0)
2022-10-20 14:39:30,601 DEBUG c.z.h.p.HikariPool: HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2022-10-20 14:39:32,198 DEBUG c.z.h.p.HikariPool: HikariPool-2 - Before cleanup stats (total=20, active=0, idle=20, waiting=0)
2022-10-20 14:39:32,198 DEBUG c.z.h.p.HikariPool: HikariPool-2 - After cleanup stats (total=20, active=0, idle=20, waiting=0)
Thanks in advance for your help
After trying to figure out whats wrong with our app, I have decided to look here for some help.
We have a Java app which uses below packages for HikariCP database pool
+--- com.custom.gwtext:gwtext-server
| +--- org.hibernate:hibernate-hikaricp:5.4.6.Final
| | +--- org.hibernate:hibernate-core:5.4.6.Final (*)
| | \--- com.zaxxer:HikariCP:3.2.0 -> 3.3.0
| | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.36
| +--- com.zaxxer:HikariCP:3.3.0 (*)
The issue is that we are currently trying to migrate to Kubernetes and we started to see a lot of java.sql.SQLException: Connection is closed errors after application is running for several hours. After enabling Hikari DEBUG log and adding leakDetectionThreshold we started to notice a few things.
This is the configuration for HikariCP
00:06:05.521 [main] DEBUG com.zaxxer.hikari.HikariConfig - HikariPool-2 - configuration:
00:06:05.522 [main] DEBUG com.zaxxer.hikari.HikariConfig - allowPoolSuspension.............false
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - autoCommit......................false
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - catalog.........................none
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - connectionInitSql...............none
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - connectionTestQuery.............none
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - connectionTimeout...............30000
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSource......................none
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceClassName.............none
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceJNDI..................none
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceProperties............{password=<masked>}
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - driverClassName................."oracle.jdbc.OracleDriver"
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - healthCheckProperties...........{}
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - healthCheckRegistry.............none
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - idleTimeout.....................600000
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - initializationFailTimeout.......1
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - isolateInternalQueries..........false
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - jdbcUrl.........................jdbc:oracle:thin:#1.1.1.1:1521:test
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - leakDetectionThreshold..........200000
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - maxLifetime.....................18000000
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - maximumPoolSize.................90
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - metricRegistry..................none
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - metricsTrackerFactory...........none
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - minimumIdle.....................15
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - password........................<masked>
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - poolName........................"HikariPool-2"
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - readOnly........................false
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - registerMbeans..................false
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - scheduledExecutor...............none
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - schema..........................none
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - threadFactory...................internal
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - transactionIsolation............default
00:06:05.528 [main] DEBUG com.zaxxer.hikari.HikariConfig - username........................"user"
00:06:05.528 [main] DEBUG com.zaxxer.hikari.HikariConfig - validationTimeout...............5000
After starting up the Java app and logging in with the user, after some minutes (probably the leakDetectionThreshold) we notice below log entry, which says that there is 2 leaks right away
[HikariPool-2 housekeeper] WARN com.zaxxer.hikari.pool.ProxyLeakTask - Connection leak detection triggered for oracle.jdbc.driver.T4CConnection#5ae99303 on thread pool-10-thread-1, stack trace follows
java.lang.Exception: Apparent connection leak detected
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:104)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:134)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:250)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:258)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246)
at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83)
at org.hibernate.internal.AbstractSharedSessionContract.beginTransaction(AbstractSharedSessionContract.java:467)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:350)
at com.sun.proxy.$Proxy326.beginTransaction(Unknown Source)
at com.custom.gwtext.server.utils.HibernateUtilImpl.startTransaction(HibernateUtilImpl.java:335)
at com.custom.gwtext.server.utils.HibernateUtilImpl.startTransactionIfNeeded(HibernateUtilImpl.java:330)
at com.custom.gwtext.server.utils.HibernateUtilImpl.getSession(HibernateUtilImpl.java:70)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.reserveIdBatch(DatabaseSequenceIdPoolDataUtil.java:29)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.reserveIdsInDatabase(DatabaseSequenceIdPool.java:106)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.access$300(DatabaseSequenceIdPool.java:24)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool$ReserveIdsTask.run(DatabaseSequenceIdPool.java:125)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
and a few minutes later second one
[HikariPool-2 housekeeper] WARN com.zaxxer.hikari.pool.ProxyLeakTask - Connection leak detection triggered for oracle.jdbc.driver.T4CConnection#7b911a8f on thread pool-6-thread-1, stack trace follows
java.lang.Exception: Apparent connection leak detected
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:104)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:134)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:250)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:258)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246)
at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83)
at org.hibernate.internal.AbstractSharedSessionContract.beginTransaction(AbstractSharedSessionContract.java:467)
at jdk.internal.reflect.GeneratedMethodAccessor166.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:350)
at com.sun.proxy.$Proxy326.beginTransaction(Unknown Source)
at com.custom.gwtext.server.utils.HibernateUtilImpl.startTransaction(HibernateUtilImpl.java:335)
at com.custom.gwtext.server.utils.HibernateUtilImpl.startTransactionIfNeeded(HibernateUtilImpl.java:330)
at com.custom.gwtext.server.utils.HibernateUtilImpl.getSession(HibernateUtilImpl.java:70)
at com.custom.gwtext.server.utils.DbUtilImpl.doWork(DbUtilImpl.java:516)
at com.custom.gwtext.server.utils.DbUtilOracleImpl.executeQueryDBSpecific(DbUtilOracleImpl.java:117)
at com.custom.gwtext.server.utils.DbUtilOracleImpl.executeQueryDBSpecific(DbUtilOracleImpl.java:67)
at com.custom.gwtext.server.utils.DbUtilImpl.executeQueryInternal(DbUtilImpl.java:266)
at com.custom.gwtext.server.utils.DbUtilImpl.executeQuery(DbUtilImpl.java:173)
at com.custom.gwtext.server.query.QueriesDefinitionRegistryImpl.getData(QueriesDefinitionRegistryImpl.java:147)
at com.custom.gwtext.server.query.QueriesDefinitionRegistryImpl.getData(QueriesDefinitionRegistryImpl.java:74)
at com.custom.pins.server.data.helper.QueryHelper.executeSelectFirstRow(QueryHelper.java:54)
at com.custom.pins.server.data.helper.QueryHelper.selectFirstRow(QueryHelper.java:36)
at com.custom.pins.server.data.helper.QueryHelper.selectFirstValueRequired(QueryHelper.java:80)
at com.custom.pins.server.business.bdt.BusinessDataTableDataUtil.existsBusinessTable(BusinessDataTableDataUtil.java:20)
at com.custom.pins.server.business.bdt.BusinessDataTableBuilderImpl.existsTable(BusinessDataTableBuilderImpl.java:225)
at com.custom.pins.server.security.audit.ws.WebServiceLoggerConfigImpl.createWsRequestConfig(WebServiceLoggerConfigImpl.java:116)
at com.custom.pins.server.security.audit.ws.WebServiceLoggerConfigImpl.getWsRequestConfig(WebServiceLoggerConfigImpl.java:75)
at com.custom.pins.server.security.audit.ws.WebServiceAuditLogService.getWsRequestSpecificConfiguration(WebServiceAuditLogService.java:88)
at com.custom.pins.server.security.audit.ws.WebServiceAuditLogService.logIfConfigEnabled(WebServiceAuditLogService.java:71)
at com.custom.pins.server.security.audit.ws.WebServiceAuditLogService.lambda$getInternalWsLogConsumer$0(WebServiceAuditLogService.java:62)
at com.custom.gwtext.server.security.audit.AuditLogServiceImpl.lambda$createEventHandler$2(AuditLogServiceImpl.java:90)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:150)
at java.base/java.lang.Thread.run(Thread.java:834)
after which they both stay active for a long time
[HikariPool-2 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-2 - After cleanup stats (total=17, active=2, idle=15, waiting=0)
Then after a few hours, this can be 3-10 hours, so no specific range, maybe depends on activity etc, no idea, this pops up in logs
[pool-10-thread-1] WARN c.zaxxer.hikari.pool.ProxyConnection - HikariPool-2 - Connection oracle.jdbc.driver.T4CConnection#5ae99303 marked as broken because of SQLSTATE(08006), ErrorCode(17002)
java.sql.SQLRecoverableException: IO Error: Connection reset by peer
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:929)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1119)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3780)
at oracle.jdbc.driver.T4CCallableStatement.executeInternal(T4CCallableStatement.java:1300)
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3887)
at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4230)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1079)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
at com.zaxxer.hikari.pool.HikariProxyCallableStatement.execute(HikariProxyCallableStatement.java)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.lambda$reserveIdBatch$0(DatabaseSequenceIdPoolDataUtil.java:34)
at org.hibernate.jdbc.WorkExecutor.executeWork(WorkExecutor.java:37)
at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2306)
at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2303)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:306)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2325)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2310)
at jdk.internal.reflect.GeneratedMethodAccessor128.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:350)
at com.sun.proxy.$Proxy326.doWork(Unknown Source)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.reserveIdBatch(DatabaseSequenceIdPoolDataUtil.java:29)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.reserveIdsInDatabase(DatabaseSequenceIdPool.java:106)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.access$300(DatabaseSequenceIdPool.java:24)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool$ReserveIdsTask.run(DatabaseSequenceIdPool.java:125)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Connection reset by peer
at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:245)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:355)
at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:144)
at oracle.net.ns.NIOHeader.readHeaderBuffer(NIOHeader.java:82)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:139)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:101)
at oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:80)
at oracle.jdbc.driver.T4CMAREngineNIO.prepareForReading(T4CMAREngineNIO.java:98)
at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:534)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:485)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612)
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:223)
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:56)
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:907)
... 29 common frames omitted
[pool-10-thread-1] INFO com.zaxxer.hikari.pool.ProxyLeakTask - Previously reported leaked connection oracle.jdbc.driver.T4CConnection#5ae99303 on thread pool-10-thread-1 was returned to the pool (unleaked)
[HikariPool-2 connection closer] DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-2 - Closing connection oracle.jdbc.driver.T4CConnection#5ae99303: (connection is broken)
Also, it's not active anymore (the active one is the second leaked one T4CConnection#7b911a8f on thread pool-6-thread-1 which somehow does not reset).
And right after this happens, our Java app starts to throw a lot of (even like 10 per second, sometimes less) below errors and becomes very slow
[pool-10-thread-1] WARN o.h.e.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: null
[pool-10-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - Connection is closed
[pool-10-thread-1] ERROR l.p.p.s.i.DatabaseSequenceIdPool - error executing work
org.hibernate.exception.GenericJDBCException: error executing work
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:47)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:311)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2325)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2310)
at jdk.internal.reflect.GeneratedMethodAccessor128.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:350)
at com.sun.proxy.$Proxy326.doWork(Unknown Source)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.reserveIdBatch(DatabaseSequenceIdPoolDataUtil.java:29)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.reserveIdsInDatabase(DatabaseSequenceIdPool.java:106)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.access$300(DatabaseSequenceIdPool.java:24)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool$ReserveIdsTask.run(DatabaseSequenceIdPool.java:125)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.sql.SQLException: Connection is closed
at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:489)
at com.sun.proxy.$Proxy307.prepareCall(Unknown Source)
at com.zaxxer.hikari.pool.ProxyConnection.prepareCall(ProxyConnection.java:290)
at com.zaxxer.hikari.pool.HikariProxyConnection.prepareCall(HikariProxyConnection.java)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.lambda$reserveIdBatch$0(DatabaseSequenceIdPoolDataUtil.java:31)
at org.hibernate.jdbc.WorkExecutor.executeWork(WorkExecutor.java:37)
at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2306)
at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2303)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:306)
... 16 common frames omitted
So, it seems like although T4CConnection#5ae99303 is marked as broken, it's never evicted and the app still tries to use it.
Can this be related to HikariCP? Looks like we are using quite outdated version of Hikari and Hibernate, but somehow those versions are baked in some custom com.custom package and not sure how we can "overwrite" those versions. We are using Gradle for building the WAR file.
Oh and also just to add, there are no such issues when running without Kubernetes. Maybe this is related to network/firewalls, but somehow I think HikariCP should work around those broken connections in case they happen.
I would really appreciate the help or even some pointers where/what to look for.
I am performing unit tests using mockito to return a dataSource using HikariDataSource,
but inform that you cannot get the connection to the network.
I understand that being a mock it is not necessary to hit the correct URL
07:24:50.913 [main] DEBUG com.zaxxer.hikari.HikariConfig - Driver class oracle.jdbc.driver.OracleDriver found in Thread context class loader sun.misc.Launcher$AppClassLoader#7106e68e
07:24:50.982 [main] WARN com.zaxxer.hikari.HikariConfig - HikariPool-1 - maxLifetime is less than 30000ms, setting to default 1800000ms.
07:24:50.982 [main] WARN com.zaxxer.hikari.HikariConfig - HikariPool-1 - idleTimeout has been set but has no effect because the pool is operating as a fixed size pool.
07:24:50.982 [main] DEBUG com.zaxxer.hikari.HikariConfig - HikariPool-1 - configuration:
07:24:50.987 [main] DEBUG com.zaxxer.hikari.HikariConfig - allowPoolSuspension.............false
07:24:50.988 [main] DEBUG com.zaxxer.hikari.HikariConfig - autoCommit......................true
07:24:50.988 [main] DEBUG com.zaxxer.hikari.HikariConfig - catalog.........................none
07:24:50.988 [main] DEBUG com.zaxxer.hikari.HikariConfig - connectionInitSql...............none
07:24:50.988 [main] DEBUG com.zaxxer.hikari.HikariConfig - connectionTestQuery.............none
07:24:50.988 [main] DEBUG com.zaxxer.hikari.HikariConfig - connectionTimeout...............20000
07:24:50.988 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSource......................none
07:24:50.988 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceClassName.............none
07:24:50.988 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceJNDI..................none
07:24:50.989 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceProperties............{password=<masked>}
07:24:50.989 [main] DEBUG com.zaxxer.hikari.HikariConfig - driverClassName................."oracle.jdbc.driver.OracleDriver"
07:24:50.989 [main] DEBUG com.zaxxer.hikari.HikariConfig - exceptionOverrideClassName......none
07:24:50.989 [main] DEBUG com.zaxxer.hikari.HikariConfig - healthCheckProperties...........{}
07:24:50.989 [main] DEBUG com.zaxxer.hikari.HikariConfig - healthCheckRegistry.............none
07:24:50.989 [main] DEBUG com.zaxxer.hikari.HikariConfig - idleTimeout.....................10000
07:24:50.989 [main] DEBUG com.zaxxer.hikari.HikariConfig - initializationFailTimeout.......1
07:24:50.990 [main] DEBUG com.zaxxer.hikari.HikariConfig - isolateInternalQueries..........false
07:24:50.990 [main] DEBUG com.zaxxer.hikari.HikariConfig - jdbcUrl.........................jdbc:oracle:thin:#192.168.55.147:1522:swd126
07:24:50.990 [main] DEBUG com.zaxxer.hikari.HikariConfig - leakDetectionThreshold..........0
07:24:50.990 [main] DEBUG com.zaxxer.hikari.HikariConfig - maxLifetime.....................1800000
07:24:50.990 [main] DEBUG com.zaxxer.hikari.HikariConfig - maximumPoolSize.................10
07:24:50.990 [main] DEBUG com.zaxxer.hikari.HikariConfig - metricRegistry..................none
07:24:50.990 [main] DEBUG com.zaxxer.hikari.HikariConfig - metricsTrackerFactory...........none
07:24:50.990 [main] DEBUG com.zaxxer.hikari.HikariConfig - minimumIdle.....................10
07:24:50.990 [main] DEBUG com.zaxxer.hikari.HikariConfig - password........................<masked>
07:24:50.990 [main] DEBUG com.zaxxer.hikari.HikariConfig - poolName........................"HikariPool-1"
07:24:50.990 [main] DEBUG com.zaxxer.hikari.HikariConfig - readOnly........................false
07:24:50.991 [main] DEBUG com.zaxxer.hikari.HikariConfig - registerMbeans..................false
07:24:50.991 [main] DEBUG com.zaxxer.hikari.HikariConfig - scheduledExecutor...............none
07:24:50.991 [main] DEBUG com.zaxxer.hikari.HikariConfig - schema..........................none
07:24:50.991 [main] DEBUG com.zaxxer.hikari.HikariConfig - threadFactory...................internal
07:24:50.991 [main] DEBUG com.zaxxer.hikari.HikariConfig - transactionIsolation............default
07:24:50.991 [main] DEBUG com.zaxxer.hikari.HikariConfig - username........................"teste"
07:24:50.991 [main] DEBUG com.zaxxer.hikari.HikariConfig - validationTimeout...............5000
07:24:50.992 [main] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
07:24:50.998 [main] WARN com.zaxxer.hikari.util.DriverDataSource - Registered driver with driverClassName=oracle.jdbc.driver.OracleDriver was not found, trying direct instantiation.
07:24:50.999 [main] DEBUG com.zaxxer.hikari.util.DriverDataSource - Driver class oracle.jdbc.driver.OracleDriver found in Thread context class loader sun.misc.Launcher$AppClassLoader#7106e68e
07:24:54.084 [main] DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to create/setup connection: IO Error: The Network Adapter could not establish the connection
07:24:54.086 [main] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Cannot acquire connection from data source
java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:445)
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:464)
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:594)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:229)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1360)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:486)
... 78 common frames omitted
#Component
public class ConfigDataSource {
#Autowired
private Properties properties;
public DataSource getDataSource(String url, String username, String password) {
DataSource dataSource = null;
try {
HikariConfig ds = new HikariConfig();
ds.setJdbcUrl(url);
ds.setUsername(username);
ds.setPassword(password);
ds.setDriverClassName(properties.getDriveClassName());
ds.setConnectionTimeout(properties.getConnectionTimeout());
ds.setMinimumIdle(properties.getMinimumIdle());
ds.setMaximumPoolSize(properties.getMaximumPoolSize());
ds.setIdleTimeout(properties.getIdleTimeout());
ds.setMaxLifetime(properties.getMaxLifetime());
dataSource = new HikariDataSource(ds);
} catch (Exception e) {
System.err.println("Error connecting to the database: " + e.getMessage());
}
return dataSource;
}
}
Mockito:
#ExtendWith(MockitoExtension.class)
public class ConfigDataSourceTest {
#Mock
private Properties properties;
#InjectMocks
private ConfigDataSource configDataSource;
#Test
public void configDataSourceTest() {
String jdbcUrl = "jdbc:oracle:thin:teste:swd001";
String userName = "teste";
String password = "teste";
String divreClassName = "oracle.jdbc.driver.OracleDriver";
Integer connectionTimeout = 20000;
Integer minimumIdle = 100;
Integer maximumPoolSize = 10;
Integer idleTimeout = 10000;
Integer maxLifetime = 1000;
when(properties.getDriveClassName()).thenReturn(divreClassName);
when(properties.getConnectionTimeout()).thenReturn(connectionTimeout);
when(properties.getMinimumIdle()).thenReturn(minimumIdle);
when(properties.getMaximumPoolSize()).thenReturn(maximumPoolSize);
when(properties.getIdleTimeout()).thenReturn(idleTimeout);
when(properties.getMaxLifetime()).thenReturn(maxLifetime);
DataSource result = configDataSource.getDataSource(jdbcUrl, userName, password);
assertNotNull(result);
}
}
I recently updated my spring-boot/spring-batch application from spring-boot-starter 1.5.9.RELEASE (uses spring-version 4.3.13.RELEASE) => 2.2.4.RELEASE (uses spring-version 5.2.3.RELEASE. The 1.5.9.RELEASE was using Hikari 2.6.3; I did try this version also in my migrated/updated 2.2.4.RELEASE version of my app. 2.2.4.RELEASE uses Hikari 3.4.2 Neither seem to work. I am using postgresql 42.2.9
This is how the Hikari logs to my console when it works (when running the 1.5.9.RELEASE spring-boot version of my app):
3c22fb53ed64 2021-04-09 05:10:58.223 WARN CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - HikariPool-1 - maxLifetime is less than 30000ms, setting to default 1800000ms.
3c22fb53ed64 2021-04-09 05:10:58.223 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - HikariPool-1 - configuration:
3c22fb53ed64 2021-04-09 05:10:58.224 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - allowPoolSuspension.............false
3c22fb53ed64 2021-04-09 05:10:58.224 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - autoCommit......................false
3c22fb53ed64 2021-04-09 05:10:58.224 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - catalog.........................none
3c22fb53ed64 2021-04-09 05:10:58.224 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - connectionInitSql...............none
3c22fb53ed64 2021-04-09 05:10:58.224 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - connectionTestQuery.............none
3c22fb53ed64 2021-04-09 05:10:58.224 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - connectionTimeout...............60000
3c22fb53ed64 2021-04-09 05:10:58.224 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - dataSource......................none
3c22fb53ed64 2021-04-09 05:10:58.224 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - dataSourceClassName.............none
3c22fb53ed64 2021-04-09 05:10:58.224 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - dataSourceJNDI..................none
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - dataSourceProperties............{password=<masked>}
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - driverClassName.................none
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - healthCheckProperties...........{}
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - healthCheckRegistry.............none
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - idleTimeout.....................600000
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - initializationFailFast..........true
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - initializationFailTimeout.......1
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - isolateInternalQueries..........false
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - jdbc4ConnectionTest.............false
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - jdbcUrl........................."jdbc:postgresql://mydatabaseurl.mywebsite.com:5432/supercooldatabase"
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - leakDetectionThreshold..........0
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - maxLifetime.....................1800000
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - maximumPoolSize.................5
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - metricRegistry..................none
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - metricsTrackerFactory...........none
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - minimumIdle.....................5
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - password........................<masked>
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - poolName........................"HikariPool-1"
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - readOnly........................false
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - registerMbeans..................false
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - scheduledExecutor...............none
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - scheduledExecutorService........internal
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - threadFactory...................internal
3c22fb53ed64 2021-04-09 05:10:58.225 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - transactionIsolation............default
3c22fb53ed64 2021-04-09 05:10:58.226 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - username........................"mktgoffers_ddladmin"
3c22fb53ed64 2021-04-09 05:10:58.226 DEBUG CorrelationId=
[main] com.zaxxer.hikari.HikariConfig - validationTimeout...............5000
3c22fb53ed64 2021-04-09 05:10:58.226 INFO CorrelationId=
[main] com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
3c22fb53ed64 2021-04-09 05:10:59.377 DEBUG CorrelationId=
[main] com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection#14ed7ddf
3c22fb53ed64 2021-04-09 05:10:59.379 INFO CorrelationId=
[main] com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Start completed.
3c22fb53ed64 2021-04-15 01:04:40.021 INFO CorrelationId=
[main] org.hibernate.dialect.Dialect - HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL82Dialect
You can see it configures Hikari with my connection information and shows the jdbc url and even adds the postgresql connection.
This is how it looks when it does NOT work, after I updated to spring-boot 2.2.4.RELEASE:
2021-04-15 01:14:11.681 INFO 75096 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2021-04-15 01:14:12.748 INFO 75096 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2021-04-15 01:14:12.760 INFO 75096 --- [ main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL82Dialect
As you can see, its different. Normally I wouldn't care about logs, but when any of my JpaRepository methods get called like .saveAll(), the application hangs/stops doing anything and never writes to database. I was perplexed after first, and thought it was my Transaction/Entity manager setup in Spring-batch, but as I said before, all my code/configuration actually works perfectly fine (writes to database) on 1.5.9.RELEASE spring boot. Now, I just discovered this database config in the log, and I'm 99% sure my database connection isn't even initializing properly.
Here is my application yaml (this is a snippet, but has all relevant info to database connection):
spring:
application:
name: My Super Cool App
profiles:
active: "myLocal,myLocalDev"
batch.job.enabled: false
datasource:
hikari:
maximumPoolSize: 5
connectionTimeout: 60000
max-lifetime: 20000
auto-commit: false
jpa:
database: POSTGRESQL
database-platform: org.hibernate.dialect.PostgreSQL82Dialect
show-sql: false
hibernate:
ddl-auto: none
properties:
hibernate:
jdbc.batch_size: 5000
order_inserts: true
flushMode: COMMIT
---
spring:
profiles: myLocal
datasource:
url: jdbc:postgresql://database-main-url.mywebsite.com:5432/cooldbname
username: mysuperkewluser
password: hahasupersecretpw
And my definition for my database configuration, DatabaseConfiguration.java:
#Configuration
#EnableTransactionManagement
public class DatabaseConfiguration {
#Autowired
private Environment envVar;
#Autowired
private KeyVault keyProvider;
#Bean(name = "dataSource")
#ConfigurationProperties(
prefix = "spring.datasource.hikari"
)
public DataSource dataSource(DataSourceProperties dataSourceProperties) {
log.debug("Configuring Datasource");
String dbPassword = keyProvider.getKeyProvider(dataSourceProperties.getPassword());
return DataSourceBuilder
.create(dataSourceProperties.getClassLoader())
.type(HikariDataSource.class)
.driverClassName(dataSourceProperties.getDriverClassName())
.url(dataSourceProperties.getUrl())
.username(dataSourceProperties.getUsername())
.password(dbPassword)
.build();
}
}
I am using this #Autowiring this dataSource in my Spring Batch JobRepository, which is defined in my BatchConfiguration.java class
#Configuration
#EnableBatchProcessing
public class BatchConfiguration {
// Autowiring of the JobBuilderFactory, StepBuilderFactory here
#Bean
public JobRepository jobRepository(PlatformTransactionManager transactionManager, #Qualifier("dataSource") DataSource dataSource) throws Exception {
JobRepositoryFactoryBean jobRepositoryFactoryBean = new JobRepositoryFactoryBean();
jobRepositoryFactoryBean.setDataSource(dataSource); // dataSource
jobRepositoryFactoryBean.setTransactionManager(transactionManager);
jobRepositoryFactoryBean.setDatabaseType("POSTGRES");
return jobRepositoryFactoryBean.getObject();
// more business logic code here
}
i have threaded system. too many (thousands)
Sometimes we have had a database bottleneck.
For analyse this bottleneck i had opened log on debug mode. And i saw this :
Hikari loaded...
springHikariCP - configuration:
allowPoolSuspension.............false
autoCommit......................true
catalog.........................none
connectionInitSql...............none
connectionTestQuery............."SELECT 1"
connectionTimeout...............30000
dataSource......................none
dataSourceClassName.............none
dataSourceJNDI..................none
dataSourceProperties............{user=blabla, url=jdbc:jtds:sqlserver:
driverClassName................."net.sourceforge.jtds.jdbc.Driver"
healthCheckProperties...........{}
healthCheckRegistry.............none
idleTimeout.....................600000
initializationFailFast..........true
initializationFailTimeout.......1
isolateInternalQueries..........false
jdbc4ConnectionTest.............false
jdbcUrl........................."jdbc:jtds:sqlserver://blabla"
leakDetectionThreshold..........0
maxLifetime.....................1800000
maximumPoolSize.................3000
metricRegistry..................none
metricsTrackerFactory...........none
minimumIdle.....................0
password........................<masked>
poolName........................"springHikariCP"
readOnly........................false
registerMbeans..................false
scheduledExecutor...............none
scheduledExecutorService........internal
threadFactory...................internal
transactionIsolation............default
username........................none
validationTimeout...............5000
springHikariCP - Starting...
and there is hundered 0 statistics until end of the log...
Pool stats (total=0, active=0, idle=0, waiting=0)
Pool stats (total=0, active=0, idle=0, waiting=0)
Pool stats (total=0, active=0, idle=0, waiting=0)
.....
How is possible it? There is too many db operations update,delete,select,insert executing. What means Total=0 connections?
Hadn't we set-load Hikari configuration properly?