Develop Reference

Camunda DB connection closed but picked by camunda engine

We are using camunda with RDS/MySql as DB. It works fine but then sometimes it says DB is closed and so throws the ProcessEngine Exception.
Here is what I understood from our config and logs:
We have 5active connections at any time in our pool (Specified in datasource config)
There was a scenario where it was closed.
We saw error like:
Request received Context path: /engine-rest Request received Path
Info: /user PathInfo: /user ExceptionHandler:
org.camunda.bpm.engine.ProcessEngineException: Process engine
persistence exception at
org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.rethrow(CommandInvocationContext.java:148)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:173)
at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:113)
at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:66)
at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)
...... Caused by: org.apache.ibatis.exceptions.PersistenceException:
Error querying database. Cause:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
No operations allowed after connection closed. The error may exist in
org/camunda/bpm/engine/impl/mapping/entity/User.xml The error may
involve
org.camunda.bpm.engine.impl.persistence.entity.UserEntity.selectUserByQueryCriteria
The error occurred while executing a query SQL: select distinct RES.*
from ACT_ID_USER RES
order by RES.ID_ asc LIMIT ? OFFSET ? Cause:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
No operations allowed after connection closed.
Our tomcat props specify: minIdle = 5;
My best guess: Its closed on server but we are maintaining locally due to the above property.
Per tomcat doc (https://tomcat.apache.org/tomcat-8.0-doc/jdbc-pool.html):
testOnBorrow = true;
validationQuery = "select 1";
The two props should fix it as it validates the connection.
Question I am trying to figure out:
How can I repro this issue? Apart from keeping connection it idle for several hours, the scenario where this happened.
Does the AWS RDS server close the connection? If so, can we control it?

As mentioned by #Zelldon, the connection timeout can be reduced and then we could try it. It works as expected.
Just to be sure, I ran two instances of camunda, one with this fix and other without. Could see that the fix worked.
Regarding RDS, it does close the connection but I could not find any documentation on it.

"java.sql.SQLException: I/O Error: Connection reset" in linux server [duplicate]

This question already has answers here:
Oracle JDBC intermittent Connection Issue
(13 answers)
Closed 5 years ago.
I have a java program which connect to oracle database sometimes it work and sometimes show me this error message :
Exception in thread "main" java.sql.SQLRecoverableException: IO Error: Connection reset
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:498)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:553)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:254)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:528)
How i can solve this problem?

Answer took from oracle forum, here:
java.security.SecureRandom is a standard API provided by sun. Among various methods offered by this class void nextBytes(byte[]).
This method is used for generating random bytes. Oracle 11g JDBC drivers use this API to generate random number during
login. Users using Linux have been encountering SQLException("Io exception: Connection reset").
The problem is two fold:
The JVM tries to list all the files in the /tmp (or alternate tmp directory set by -Djava.io.tmpdir) when
SecureRandom.nextBytes(byte[]) is invoked. If the number of files is large the
method takes a long time
to respond and hence cause the server to timeout
The method void nextBytes(byte[]) uses /dev/random on Linux and on some machines which lack the random
number generating hardware the operation slows down to the extent of bringing the whole login process to
a halt. Ultimately the the user encounters SQLException("Io exception:
Connection reset")
Users upgrading to 11g can encounter this issue if the underlying OS is Linux which is running on a faulty hardware.
The cause of this has not yet been determined exactly. It could either be a problem in the hardware or the fact
that for some reason the software cannot read from dev/random
a solution seems to add this setting to the jvm
-Djava.security.egd=file:/dev/./urandom

Connection reset usually happens when the connection between you app and the database gets disconnected for example network issue or resources that's required.
Check the following post it should give you an idea of what to check.
SQLRecoverableException: I/O Exception: Connection reset

JDBC Connection Pool: connections are not recycled after DB restart

I added setMaxActive(8) on org.apache.tomcat.jdbc.pool.PoolProperties. Every time the DB restarts, the application is unusable because the established connections remain. I get the following error:
org.postgresql.util.PSQLException: This connection has been closed
I've tried using some other settings on the pool to no avail...
Thank you for help!

Use the validationQuery property which will check if the connection is valid before returning the connection.
Ref: Tomcat 6 JDBC Connection Pool
This property is available on latest tomcat versions.

Look at this link:
Postgres connection has been closed error in Spring Boot
Very valid question and this problem is usually faced by many. The
exception generally occurs, when network connection is lost between
pool and database (most of the time due to restart). Looking at the
stack trace you have specified, it is quite clear that you are using
jdbc pool to get the connection. JDBC pool has options to fine-tune
various connection pool settings and log details about whats going on
inside pool.
You can refer to to detailed Apache documentation on pool
configuration to specify abandon timeout
Check for removeAbandoned, removeAbandonedTimeout, logAbandoned parameters
Additionally you can make use of additional properties to further
tighten the validation
Use testXXX and validationQuery for connection validity.
My own $0.02: use these two parameters:
validationQuery=<TEST SQL>
testOnBorrow=true

How to reconnect when the LDAP server is restarted?

I have a situation where through a Java program, I create a javax.naming.ldap.LdapContext and do a search() operation on it - which makes an underlying connection. Then I put the Java app thread to sleep, during which I restart the LDAP server (OpenLDAP, just to note). When the App thread wakes up and tries to do any operation on the LdapContext created earlier, it throws "CommunicationException: Connection is closed".
What I want is to be able to re-establish the connection.
I see that LdapContext has a reconnect() method - where I pass controls as null. However, this does not have any effect. What I saw in the Sun LDAP implementation that during the time when the LDAP server was restarted, the ConnectionPool maintained by the Sun implementation marked the underlying com.sun.jndi.ldap.LdapClient instance with a "usable=false". Upon reconnect() call - it simply calls ensureOpen(), which again checks if the usable flag is false or not - if it's false; then it throws CommunicationException - so back to square one.
My question is: how does a Java app survive an external LDAP server restart? Is creation of new LdapContext again is the only way out?
Appreciate any insights.
Here is the stacktrace of the exception:
javax.naming.CommunicationException: connection closed [Root exception is java.io.IOException: connection closed]; remaining name 'uid=foo,ou=People,dc=example,dc=com'
at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1979)
at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1824)
at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1749)
at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:368)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:338)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:321)
at javax.naming.directory.InitialDirContext.search(InitialDirContext.java:248)
Caused by: java.io.IOException: connection closed
at com.sun.jndi.ldap.LdapClient.ensureOpen(LdapClient.java:1558)
at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:504)
at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1962)
... 26 more

Just enable JNDI connection pooling and it will all be taken care of for you behind the scenes. See the JNDI Guide to Features and the LDAP Provider documentation. It's controlled by just a couple of properties.

The UnboundID LDAP SDK provides a means to auto-connect wherein that auto-reconnect operation is invisible to the client.

We had this problem at work. The solution we came up with (may not be the best answer). Was to create a watchdog thread that would check the connection at some fixed rate. If the connection did not work, it would re-initialize the connection with LDAP.

You should note that this is related essentially to LDAP connection pooling. As defined here:
A connection is retrieved from the pool, used, returned to the pool, and then, retrieved again from the pool for another Context instance.
Thus, the reuse of a previous connection may cause such problem:
You may test the behavior without using LDAP connection pooling by setting
com.sun.jndi.ldap.connect.pool=false
Also, another possible cause may be the timeout of reading the LDAP operations. In fact, the reading operation is not notified about the closure of the LDAP server after a specific timeout. For more information, you may take a look at this link

How to remove invalid database connection from pool

I am using connection pooling of tomcat with oracle database. It is working fine, but when i use my application after a long time it is giving error that "connection reset". I am getting this error because of physical connection at oracle server closed before logical connection closed at tomcat datasource. So before getting the connection from datasource i am checking the connection validity with isValid(0) method of connection object which gives false if the physical connection was closed. But i don't know how to remove that invalid connection object from the pool.

This could be because on the db server, there is a timeout to not allow connections to live beyond a set time, or to die if it does not receive something saying it is still valid. One way to fix this is to turn on keepalives. These basically ping the db server saying that they are still valid connections.
This is a pretty good link on Tomcats DBCP configurations. Take a look at the section titled "Preventing dB connection pool leaks". That looks like it may be a good place to start.

I used validatationquery while configuring the datasource in server.xml file. It is going to check the validity of the connection by executing the query at database before giving to the application.
for Oracle
validationQuery="/* select 1 from dual */"
for MySql
validationQuery="/* ping */"

Try closing it and opening it if it's invalid. I mean u would reinitialize it in this way so u won't need to remove it from the pool and reuse it.

If we want to dispose an ill java.sql.connection from Tomcat jdbc connection pool,
we may do this explicitly in the program.
Unwrap it into an org.apache.tomcat.jdbc.pool.PooledConnection,
setDiscarded(true) and close the JDBC connection finally.
The ConnectionPool will remove the underlying connection once it has been returned.
(ConnectionPool.returnConnection(....))
e.g.
PooledConnection pconn = conn.unwrap(PooledConnection.class); pconn.setDiscarded(true);
conn.close();