We are trying to integrate the HikariCP (release 2.4.4) into our application. After some time of usage the pool fails to acquire new connections throwing:
java.lang.NullPointerException
at com.zaxxer.hikari.pool.PoolBase.setNetworkTimeout(PoolBase.java:464) ~[HikariCP-2.4.4.jar:?]
at com.zaxxer.hikari.pool.PoolBase.isConnectionAlive(PoolBase.java:131) ~[HikariCP-2.4.4.jar:?]
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:171) ~[HikariCP-2.4.4.jar:?]
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:147) ~[HikariCP-2.4.4.jar:?]
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83) ~[HikariCP-2.4.4.jar:?]
The jdbc driver we are using is ojdbc-7 with version 12.1.0.2. The pool uses following configuration:
allowPoolSuspension.............false
autoCommit......................false
catalog.........................null
connectionInitSql..............."BEGIN EXECUTE IMMEDIATE 'SET ROLE SOME_ROLE IDENTIFIED BY SOME_PASSWORD '; END;"
connectionTestQuery............."SELECT 1 FROM DUAL"
connectionTimeout...............15000
dataSource......................null
dataSourceClassName.............null
dataSourceJNDI..................null
dataSourceProperties............{v$session.machine=host, password=<masked>, v$session.program=my application}
driverClassName................."oracle.jdbc.OracleDriver"
healthCheckProperties...........{}
healthCheckRegistry.............null
idleTimeout.....................60000
initializationFailFast..........true
isolateInternalQueries..........false
jdbc4ConnectionTest.............false
jdbcUrl........................."jdbc:oracle:thin:#//host.company.com:1521/database.company.com"
leakDetectionThreshold..........1800000
maxLifetime.....................0
maximumPoolSize.................18
metricRegistry..................null
metricsTrackerFactory...........null
minimumIdle.....................2
password........................<masked>
poolName........................"TEST_POOL"
readOnly........................false
registerMbeans..................false
scheduledExecutorService........null
threadFactory...................null
transactionIsolation............null
username........................"USER_NAME"
validationTimeout...............5000
Is it a bug or a missconfiguration?
I'm not 100% sure, but it looks like either:
You are evicting a connection after you have called close(), which is not allowed. Or..
You are evicting a connection and then calling close(), which is not allowed.
When you evict a connection, you must be the owner of that connection (obtained from getConnection(), and subsequently you must not close() the connection (it will be closed automatically). And as explained above, if you have called close() already, the connection is already back in the pool and it is not valid to evict it, as you are no longer the owner.
EDIT: Let me be clearer. From studying how this exception could be reached it seems clear that you are first closing the connection, and secondly evicting the connection. The reverse (evict and then close) would not result in this error.
This is known as "use after return" and is similar to "use after free" bugs in languages without garbage collection. When you close a connection, it is returned to the pool. From that instant the connection is available to be claimed by another thread -- the caller of close() is no longer the owner.
This is exactly analogous to calling free() on memory in C/C++. Instantly after doing so the memory is available to be claimed by another caller -- the caller of free() is no longer the owner. In the C/C++ case, if you continue to use a reference to the freed memory you risk corrupting data of another thread that has now allocated it.
In the case of nearly any pooling library in Java (connection or otherwise), once you release an object back to the pool you are no longer the owner. Nothing can prevent you from retaining a reference to the returned object.
In this case, once you have called close(), the object is returned to the pool instantly. If another thread obtains the connection legally from the pool (getConnection()), while at the same time the previous owner calls evict(), you will easily run into this issue.
We may choose to harden this code path (or not). HikariCP is not particularly paternalistic philosophically; favoring documentation over code. For example, if you pass a null into evict() you will be met with an NPE somewhere. Could we check for null and ignore it? Sure. Multiply that approach across the codebase and it could easily grow by 20%. Or, how about don't do that, developer?
It is a fairly simple contract:
You can only evict a connection that you own.
As soon as you have closed a connection, you no longer own it.
Related
How to reset a JDBC Connection object (that is, a java.sql.Connection object)?
I have explored the concept of connection pooling. When a connection pool is used, a Connection object can be recycled. But how can a connection pool recycle a Connection object? I think that a connection needs to be "reset" (for example, if it is in a transaction, then perhaps rollback it, but there may be more things to reset) before it can be reused. But I cannot find such a "reset" method in the Java documentation about the class java.sql.Connection.
If you are talking about what you as a user of a connection should do, then that is simple: call close() on the connection. Closing the logical connection will signal to the connection pool that the connection is available for reuse, and the connection pooling manager is then responsible for performing the necessary reset, invalidation of statement handles, etc.
If you are talking about what you as the implementer of a connection pooling manager should do, that is where things become complicated.
Historically, JDBC provides no way to 'reset' a java.sql.Connection, other than by your own code (or your third-party connection pool) remembering the initial configuration, and restoring it after use, and keeping track of objects like statements and result sets and closing them when the connection is returned to the pool.
Originally, the intended way to reset connections in JDBC was for an application server to use a driver's javax.sql.ConnectionPoolDataSource as a factory for javax.sql.PooledConnection objects. These PooledConnection objects serve as handles for physical connections to be held in a connection pool (to be clear, ConnectionPoolDataSource is not a connection pool, it is a data source for a connection pool). The application server would then expose a javax.sql.DataSource handing out logical Connection objects, where on Connection.close(), the driver specific implementation of PooledConnection would take care of any necessary reset of a connection (though JDBC underspecifies what is 'necessary').
However, in practice this route is hardly ever used because support in drivers was (and often still is) spotty, inconsistent or downright incorrect, and JDBC wasn't clear enough on exactly what needed to be done when the logical connection was closed. The world has also shifted to using third-party connection pool libraries that do not use ConnectionPoolDataSource.
JDBC 4.3 (Java 9 and higher) introduced the methods Connection.beginRequest() and Connection.endRequest() to be called by connection pooling managers, which would seem to fit such pattern, but unfortunately JDBC 4.3 doesn't actually specify what kind of things an implementation should or should not do in response to beginRequest and endRequest.
In short, there is no real general way to reset a connection in JDBC.
Do you try to close() method and reinitialize Connection object again.
close()
Releases this Connection object's database and JDBC resources immediately instead of waiting for them to be automatically released
The java application using ojdbc6.jar, tomcat 7, tomcat-dbcp-8.0.3.jar (and other jars that's probably not relevant to this question), JDK 7 (u51)
We have identified that there is a connection leak using v$session report where some connection gets into INACTIVE state for 7+ hours. This is also confirmed by Thread dump taken at frozen state.
Heap Dump (taken at frozen state) shows:
Total PoolableConnection and DefaultPooledObject is equal to maxTotal (expected for exhausted pool)
Each connection is associated to PooledObjectState ALLOCATED
(expected)
lastReturnTime = lastUserTime = lastBorrowedTime (in
DefaultPooledObject), which to me means: THREAD-1 (good workflow)
returned the connection which was immediately borrowed by THREAD-2
(bad workflow having leak) and THREAD-2 never closed the
connection, let it dangling!
ALL of the above observation makes sense, since we definitely have connection leak and eventual exhausted pool
My question is:
When I see, details about PoolableConnection, it has associated boolean _closed which is "true". Why/How could it have "_closed = true".
when I decompiled tomcat-dbcp jar, I could see that every time _closed is marked true, it will also associate IDLE state to connection object (instead of ALLOCATED).
Looking for theories on why this boolean is true.
PS: We have various ideas (like setting logAbandoned) to find the exact piece of code responsible for connection leak, I am looking forward to find a reason (or theory) for heap dump to capture these PoolableConnection _closed=true case.
Looking at DelegatingConnection source code it can be seen that closed=true can be set as the result of connection.close() or in a finally block after some exceptions as a safety measure.
} finally {
closed = true;
}
There's a leak, the connection is in an inconsistent state because it could not be closed and is probably ready to be processed in the ABANDONED life cycle phase.
Inspecting the pool through JMX could give another perspective.
The leak could be related to an improperly handled exception that otherwise would have given a hint on the pool bad state.
I asked this question (How do I call java.sql.Connection::abort?) and it led me to another question.
With
java.sql.Connection conn = ... ;
What is the difference between
conn.close();
and
conn.abort(...);
?
You use Connection.close() for a normal, synchronous, close of the connection. The abort method on the other hand is for abruptly terminating a connection that may be stuck.
In most cases you will need to use close(), but close() can sometimes not complete in time, for example it could block if the connection is currently busy (eg executing a long running query or update, or maybe waiting for a lock).
The abort method is for that situation: the driver will mark the connection as closed (hopefully) immediately, the method returns, and the driver can then use the provided Executor to asynchronously perform the necessary cleanup work (eg making sure the statement that is stuck gets aborted, cleaning up other resources, etc).
I hadn't joined the JSR-221 (JDBC specification) Expert Group yet when this method was defined, but as far as I'm aware, the primary intended users for this method is not so much application code, but connection pools, transaction managers and other connection management code that may want to forcibly end connections that are in use too long or 'stuck'.
That said, application code can use abort as well. It may be faster than close (depending on the implementation), but you won't get notified of problems during the asynchronous clean up, and you may abort current operations in progress.
However keep in mind, an abort is considered an abrupt termination of the connection, so it may be less graceful than a close, and it could lead to unspecified behaviour. Also, I'm not sure how well it is supported in drivers compared to a normal close().
Consulting the java docs seems to indicate that abort is more thorough than close, which is interesting.
abort...
Terminates an open connection. Calling abort results in: The
connection marked as closed Closes any physical connection to the
database Releases resources used by the connection Insures that any
thread that is currently accessing the connection will either progress
to completion or throw an SQLException.
close...
Releases this Connection object's database and JDBC resources
immediately instead of waiting for them to be automatically released.
Calling the method close on a Connection object that is already closed
is a no-op.
So it seems if you are only concerned with releasing the objects, use close. If you want to make sure it's somewhat more "thread safe", using abort appears to provide a more graceful disconnect.
Per Mark Rotteveel's comment (which gives an accurate summary of the practical difference), my interpretation was incorrect.
Reference: https://docs.oracle.com/javase/8/docs/api/java/sql/Connection.html#close--
I am trying to use HikariCP connection pool. I was able to get it to work and get a connection that I could use. I am not sure what is the best approach for returning the connection to the pool.
I have the following questions:
Should I close the connection when I am done, rely on idleTimeout
and maxLifetime settings or is there another call that I can use so
as not to hog the connections from the pool?
If I close the connections (instead of returning to the pool), would
that not result in additional connection objects being created
to meet the requirements of the connection pool size?
Looking for helpful suggestions.
As with most connection pools, Hikari doesn't give you an actual JDBC Connection when you ask for one. What it does instead is give you a proxy that implements the Connection interface. In the case of Hikari - it's a ConnectionProxy object.
This proxy serves a few purposes, the main of which is - take the control of opening/closing connections and statements away from you and into the connection pool. This happens automagically and you should be using your connections as usual. This includes closing them after use.
If you look at the source code for Hikari, at the ConnectionProxy class in particular, you will see that the close() method is very different from the standard one. The code reads as:
Mark the connection as closed, do cleanup, reset underlying connection state and params.
Hence, simply calling close() will just clean and return the connection to the pool.
We have seen connection droughts in our system every once in a while, and the problem seems to be that Sessions are not being returned to the connection pool quick enough. I wrote a test that seems to confirm using Session.disconnect() on the sessions (after being done with one) will solve this problem. However, I also timed these calls, and it seems like using disconnect is increasing running time by 3 times.
According to the docs (http://docs.jboss.org/hibernate/core/3.5/api/org/hibernate/Session.html#disconnect() ), disconnect should be returning it to the connection pool. However, the doc also says it "closes" the connection. I'm not sure what it means because I know for a fact that Session.close() does more than disconnect, and what good would a connection pool be if you close the connection before returning it?
In any case, I'm wondering why a method that returns the session to the connection pool would be anything but instantaneous and essentially free. Surely thats the whole point of a connection pool, right?
Any ideas would be appreciated.