Develop Reference

Closed connection issues after DB upgrade from 11G to 19C & ojdbc14 to ojdbc8

I managed to successfully perform a DB upgrade from 11G to 19C & ojdbc14 to ojdbc8. However my application is now facing closed connection issues, java.sql.SQLRecoverableException: Closed Connection for almost all my queries after every few days. This issue usually go away after I perform a server restart. I am running my application on JBOSS WildFly 13.
I noticed this usually occurs when i ran this query, and there are around 170 records. However most of the Status are inactive. I am thinking this could be a out of resource memory issue. Instead of doing a server restart every few days, how am I supposed to fix this issue to remove the inactive status completely? Could this be a leak within my Java application, I have made very minimal code changes to the aplication except for some autocommit changes which were causing errors..
select
substr(a.spid,1,9) pid,
substr(b.sid,1,5) sid,
substr(b.serial#,1,5) ser#,
substr(b.machine,1,6) box,
substr(b.username,1,10) username,
substr(b.osuser,1,8) os_user,
substr(b.program,1,30) program,
status
from v$session b, v$process a
where
b.paddr = a.addr
and type='USER'
order by spid;

Spring Boot application giving JDBC connection not found error quite often

My Spring boot application is using Spring Data JPA to interact with a AWS MySQL RDS. The application is deployed on 2 ec2 instances on top of ELB.
MySQL DB has a maximum connection limit of 147 connections, so I have added following entry in my application-prod.properties file:
spring.datasource.hikari.connection-timeout=30000
spring.datasource.hikari.maximum-pool-size=70
After this my application runs fine for few days but then I again start getting JDBC connection not found error.
On further debugging running following command on my MySQl gives me only 73 active connections whereas for the time when application was running properly, it returned 143.
Could it be possible that 1 of the ec2 instance is stopped or terminated?
If so what can I do to debug further?
SHOW STATUS WHERE `variable_name` = 'Threads_connected';
I have had a hard time to debug this issue, need some help.

Camunda DB connection closed but picked by camunda engine

We are using camunda with RDS/MySql as DB. It works fine but then sometimes it says DB is closed and so throws the ProcessEngine Exception.
Here is what I understood from our config and logs:
We have 5active connections at any time in our pool (Specified in datasource config)
There was a scenario where it was closed.
We saw error like:
Request received Context path: /engine-rest Request received Path
Info: /user PathInfo: /user ExceptionHandler:
org.camunda.bpm.engine.ProcessEngineException: Process engine
persistence exception at
org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.rethrow(CommandInvocationContext.java:148)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:173)
at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:113)
at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:66)
at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)
...... Caused by: org.apache.ibatis.exceptions.PersistenceException:
Error querying database. Cause:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
No operations allowed after connection closed. The error may exist in
org/camunda/bpm/engine/impl/mapping/entity/User.xml The error may
involve
org.camunda.bpm.engine.impl.persistence.entity.UserEntity.selectUserByQueryCriteria
The error occurred while executing a query SQL: select distinct RES.*
from ACT_ID_USER RES
order by RES.ID_ asc LIMIT ? OFFSET ? Cause:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
No operations allowed after connection closed.
Our tomcat props specify: minIdle = 5;
My best guess: Its closed on server but we are maintaining locally due to the above property.
Per tomcat doc (https://tomcat.apache.org/tomcat-8.0-doc/jdbc-pool.html):
testOnBorrow = true;
validationQuery = "select 1";
The two props should fix it as it validates the connection.
Question I am trying to figure out:
How can I repro this issue? Apart from keeping connection it idle for several hours, the scenario where this happened.
Does the AWS RDS server close the connection? If so, can we control it?

As mentioned by #Zelldon, the connection timeout can be reduced and then we could try it. It works as expected.
Just to be sure, I ran two instances of camunda, one with this fix and other without. Could see that the fix worked.
Regarding RDS, it does close the connection but I could not find any documentation on it.

Hikari: Failed to validate connection because connection is closed

I'm using hikari pool connection through play framework and mariadb client and since I've updated them (play 2.6.5 -> 2.6.6 and mariadb 2.1.1 -> 2.1.2 but not sure it's related) regularly I've got the following error:
HikariPool-1 - Failed to validate connection org.mariadb.jdbc.MariaDbConnection#31124a47 (Connection.setNetworkTimeout cannot be called on a closed connection)
at com.zaxxer.hikari.pool.PoolBase.isConnectionAlive(PoolBase.java:184)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:172)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:85)
at play.api.db.DefaultDatabase.getConnection(Databases.scala:142)
at play.api.db.DefaultDatabase.withConnection(Databases.scala:152)
at play.api.db.DefaultDatabase.withConnection(Databases.scala:148)
I've found a related issue here and tried to change the idleTimeout and maxLifetime to 2 and 5 minutes but the error still happened.
I'm using HikariCP 2.7.1, play 2.6.6 and mariadb-java-client 2.1.2

Although you write that you had no success solving this issue by changing the maxLifetime value, I wanted to note that it actually worked for me. Putting its value to 590000 has removed the warnings from my log file.
The maxLifetime (in milliseconds) value of your client should be less than the wait_timeout (in seconds) value of your MySQL instance. This way the client will always terminate the connection before the database tries to. The other way around, the client will try to act upon a closed connection and you will get the above mentioned warnings in your log file.
To see the wait_timeout value of your MySQL instance, you can use the following query:
SHOW VARIABLES like '%timeout%';
The default maxLifetime value for MariaDB should be 28800, but I noticed that 600 can be in place because of MySQL config files being loaded.
I should note that I have no other explicit hikari configuration in place except for a maximum-pool-size of 50.
I got the inspiration from: https://github.com/brettwooldridge/HikariCP/issues/856 by the way. Other very useful resources are: https://github.com/brettwooldridge/HikariCP#configuration-knobs-baby and https://mariadb.com/kb/en/library/server-system-variables/#wait_timeout

Grails : Undeploy on Glassfish causes connection pool to be no more usable

I'm working on a grails 2.2.1 project running on glassfish 3.1.2.2, using a jndi pooled datasource connected to a postgres 9.2 database.
When I deploy for the first time the application, I have no issues, everything works fine.
But, everytime I undeploy (or redeploy) my application, my connection pool stops working.
If I try to ping using glassfish admin console, I get this error :
Avertissement: RAR8054: Exception while creating an unpooled [test] connection for pool [ sample ],
Connection could not be allocated because: No suitable driver found for
jdbc:postgresql://localhost:5432/sample?loginTimeout=0&socketTimeout=0&prepareThreshold=5&unknownLength=2147483647&loglevel=0&tcpkeepalive=false
Grave: RestResponse.getResponse() donne FAILURE.
endpoint = http://localhost:4848/management/domain/resources/ping-connection-pool.json ; attrs = {id=sample}'
I repeat : my connection pool works fine when I deploy (I have the postgresql driver in "/glassfish/lib", my connection pool is correctly configured, etc.), it's only when I undeploy that I have troubles.
This bug is highly reproductible, I even did a small grails project that reproduces the problem, you only need to have to configure a jndi datasource connected to a "sample" database (jndi/sample) on your glassfish server.
Here is the github link :
https://github.com/fabier/GlassfishJDBCGrailsPostgresql
And the war :
https://www.dropbox.com/s/cb9hmgc6lfabl5x/GlassfishJDBCGrailsPostgresql-0.1.war
and the domains.xml file:
https://www.dropbox.com/s/ljealpfp0zxj2r4/domain.xml
On grails side, my connection to the pool is configured this way:
environments {
production {
dataSource {
dbCreate = "create-drop"
jndiName = "jdbc/sample"
}
}
}
I think this bug is severe, since it kills a connection pool that can be shared amongst other webapps or ears.
For information, I did a bit of research, sensing it might be some sort of class unloader associated with undeployment, I found log4j that could be linked to this problem, so I tryed :
-Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false
but it didn't help.
Last point : in fact, postgresql driver is fully unloaded, so that other connection pools using postgresql driver are not usable after an undeploy.
Link to same issue in Grails mailing List:
http://grails.1312388.n4.nabble.com/Undeploy-on-Glassfish-causes-connection-pool-to-be-no-more-usable-td4643041.html
Link to same issue in Glassfish's Jira:
http://java.net/jira/browse/GLASSFISH-20094
Thanks for your help.