I use JDBC pool for my Tomcat web applications.
As I read in docs, I need get connection, execute query and close.
After close, connection returns to pool.
I think Way 1 better when query DB depend on external event and for the task which running every 5sec Way 2 more correct.
Could someone explain which way use for task which repeat every 5 sec?
PS: I skip extra checks in code to make code looks readable.
Way #1 Gets connection from pool and close every 5 sec
Connection c = null;
Statement s = null;
ResultSet rs = null;
DataSource ds = ... Get DataSource ...
while(running) {
try {
c = ds.getConnection();
s = c.createStatement();
rs = s.executeQuery('SELECT data FROM my_table');
... do something with result ...
} catch (SQLException sec) {
... print exception ...
} finally {
try {
rs.close();
s.close();
c.close();
} catch (SQLException sec) { ... print exception ... }
... Thread sleep 5 seconds and repeat ...
}
}
Way #2 Get connection before loop and close after, reconnect inside loop
Connection c = null;
Statement s = null;
ResultSet rs = null;
DataSource ds = ... Get DataSource ...
c = ds.getConnection();
while(running) {
try {
s = c.createStatement();
rs = s.executeQuery('SELECT data FROM my_table');
... do something with result ...
} catch (SQLException sec) {
... print exception ...
... if connection lost, try reconnect and execute query again ...
} finally {
try {
rs.close();
s.close();
} catch (SQLException sec) {
... print exception ...
}
... Thread sleep 5 seconds and repeat ...
}
}
c.close();
Pool config
<Resource name="jdbc/pg_mega" auth="Container"
type="javax.sql.DataSource" driverClassName="org.postgresql.Driver"
url="jdbc:postgresql://127.0.0.1:6432/db"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
username="***" password="****"
defaultAutoCommit="true"
initialSize="1"
maxActive="300"
maxTotal="300"
maxIdle="20"
minIdle="5"
maxWait="10000"
validationQuery="select 1"
validationInterval="30000"
testWhileIdle="false"
testOnBorrow="true"
testOnReturn="false"
timeBetweenEvictionRunsMillis="30000"
minEvictableIdleTimeMillis="30000"
/>
I think the most common pattern is this:
Connection conn = null;
PreparedStatement stmt = null;
ResultSet res = null;
try {
conn = ds.getConnection();
stmt = conn.prepareStatement(sqlStatement);
//
// ....
res = stmt.executeQuery();
// use the resultset
conn.commit();
} catch (SQLException e) {
// Manage the exception
try {
conn.rollback();
} catch (SQLException e1) {
// SWALLOW
}
} finally {
close(res);
close(stmt);
close(conn);
}
I use these helper functions to safely close without too much boilerplate, from java 7 on you can autoclose so these helper are no use anymore.
public static void close(Connection conn) {
try {
if (conn != null)
conn.close();
} catch (SQLException e) {
// SWALLOW
}
}
public static void close(Statement stmt) {
try {
if (stmt != null)
stmt.close();
} catch (SQLException e) {
// SWALLOW
}
}
public static void close(ResultSet res) {
try {
if (res != null)
res.close();
} catch (SQLException e) {
// SWALLOW
}
}
You should really be sure to close the connection in the finally statement, bad things happens if you do not close the connection, and if (for example) the rs is null in your example (not so difficult) you won't close the connection.
Getting and releasing the connection from the pool is not a performance problem, it happens in microseconds , thousands of times faster then any possible query.
The reason why you do not release the connection eagerly is transactions, you want to keep the same connection for the whole transaction (no way around this).
When you commit (or rollback) then you don't need that peculiar connection anymore, so just release it.
Another hint, close the connection in finally, even if you catch SQL Exceptions, because there are always Runtime Exceptions and even Errors (which you will not catch), but finally will be executed even in the face of OutOfMemoryError or ClassDefNotFoundError or any other, and the connection will be returned to the pool.
Last but not least is the pool that will try to reconnect in case of disconnection, in fact the pool will just ditch invalid connection and create a fresh batch when needed.
You should choose a good policy of connection validation, a bad choice will lead to much extra time in getting the connection thus hitting hard on performance, or to exceptions caused by invalid connection acquired from the pool.
Optimizing the pool is like many other performance tuning tasks: HARD .
For example:
testOnBorrow="true" will hit the DB before acquiring the connection, it is safe, but it will cost, tens or hundreds of time slower that not checking on borrow.
testWhileIdle="true" instead is less safe (you could get an invalid connection) but is much faster an has the advantage of keeping alive the connections.
You have to choose considering how you use connection, how you deal with errors, where is the DB (on the same machine, on a lan, on a wan) and many other factors.
Way #2 is not correct when you use a pool. If you use a pool, you should always try to keep the connection out of the pool ("leased") for as short as possible to get the most out of pool usage. If you do not use a pool, you have to consider the cost of creating and destroying connections and manage the connection's life cycle.
If you use a pool, a leased connection must always be returned to the pool (a connection is returned to the pool when you close the connection). If connections are not returned to the pool (i.e. connections are leaked), the pool will be empty soon and your application will stop working. This is especially important when things go wrong (e.g. in your code example, when rs is null due to a query error, the connection will be leaked). To prevent connections from leaking, consider using a tool like Sql2o which has built-in protection against connection leakage.
Also reduce the number of connections in the pool. Start with minIndle="1" and maxActive="4". Use stress-testing to determine the upper-limit of the pool-size (more connections in a pool usually do more harm than good, see also About Pool Sizing from HikariCP which has more good articles about database connection pools).
Related
We use connection pool in our application. While I understand that we should close and get connections as needed since we are using a connection pool. I implemented a cache update mechanism by receiving Postgres LISTEN notifications. The code is pretty much similar to the canonical example given by the documentation.
As you can see in the code, the query is initiated in the constructor and the connection is re used. This may pose problem when the connection is closed out of band due to any factor. One solution to this is to get the connection before every use, but as you can see the statement is only executed once in the constructor but still I can receive the notification in the polling. So if I get the connection every time, it will force me to re issue the statement for every iteration(after delay). I'm not sure if that's an expensive operation.
What is the middle ground here?
class Listener extends Thread
{
private Connection conn;
private org.postgresql.PGConnection pgconn;
Listener(Connection conn) throws SQLException
{
this.conn = conn;
this.pgconn = conn.unwrap(org.postgresql.PGConnection.class);
Statement stmt = conn.createStatement();
stmt.execute("LISTEN mymessage");
stmt.close();
}
public void run()
{
try
{
while (true)
{
org.postgresql.PGNotification notifications[] = pgconn.getNotifications();
if (notifications != null)
{
for (int i=0; i < notifications.length; i++){
//use notification
}
}
Thread.sleep(delay);
}
}
catch (SQLException sqle)
{
//handle
}
catch (InterruptedException ie)
{
//handle
}
}
}
In addition to this, there is also another similar document which had another query in run method as well in addition to constructor. I'm wondering if someone could enlighten me the purpose of another query within the method.
public void run() {
while (true) {
try {
//this query is additional to the one in the constructor
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery("SELECT 1");
rs.close();
stmt.close();
org.postgresql.PGNotification notifications[] = pgconn.getNotifications();
if (notifications != null) {
for (int i=0; i<notifications.length; i++) {
System.out.println("Got notification: " + notifications[i].getName());
}
}
// wait a while before checking again for new
// notifications
Thread.sleep(delay);
} catch (SQLException sqle) {
//handle
} catch (InterruptedException ie) {
//handle
}
}
}
I experimented closing the connection in every iteration(but without getting another one). That's still working. Perhaps that's due to unwrap that was done.
Stack:
Spring Boot, JPA, Hikari, Postgres JDBC Driver (not pgjdbc-ng)
The connection pool is the servant, not the master. Keep the connection for as long as you are using it to LISTEN on, i.e. ideally forever. If the connection ever does close, then you will miss whatever notices were sent while it was closed. So to keep the cache in good shape, you would need to discard the whole thing and start over. Obviously not something you would want to do on a regular basis, or what would be the point of having it in the first place?
The other doc you show is just an ancient version of the first one. The dummy query just before polling is there to poke the underlying socket code to make sure it has absorbed all the messages. This is no longer necessary. I don't know if it ever was necessary, it might have just been some cargo cult that found its way into the docs.
You would probably be better off with the blocking version of this code, by using getNotifications(0) and getting rid of sleep(delay). This will block until a notice becomes available, rather than waking up twice a second and consuming some (small) amount of resources before sleeping again. Also, once a notice does arrive it will be processed almost immediately, instead of waiting for what is left of a half-second timeout to expire (so, on average, about a quarter second).
I have been learning about using MySQL within Java using Oracle JDBC and I am trying to get into the mindset of try/catch and pool cleanup.
I am wondering if the following code is the correct way to perfectly clean everything up or if you notice holes in my code that requires something I've missed. For the record, I intend to use InnoDB and its row locking mechanism which is why I turn auto commit off.
try
{
connection = getConnection(); // obtains Connection from a pool
connection.setAutoCommit(false);
// do mysql stuff here
}
catch(SQLException e)
{
if(connection != null)
{
try
{
connection.rollback(); // undo any changes
}
catch (SQLException e1)
{
this.trace(ExtensionLogLevel.ERROR, e1.getMessage());
}
}
}
finally
{
if(connection != null)
{
try
{
if(!connection.isClosed())
{
connection.close(); // free up Connection so others using the connection pool can make use of this object
}
}
catch (SQLException e)
{
this.trace(ExtensionLogLevel.ERROR, e.getMessage());
}
}
}
getConnection() returns a Connection object from a pool and connection.close() closes it releasing it back to the pool (so I've been told, still new to this so apologies if I am talking rubbish). Any help on any of this would be greatly appreciated!
Thank you!
I recommend not setting autocommit back to true in the finally block - your other threads that are relying on autocommit being set to true should not assume that the connections in the pool are in this state, but instead they should set autocommit to true before using a connection (just as this thread is setting autocommit to false).
In addition, you should check the connection's isClosed property before calling close() on it.
Other than that, I don't see any problems.
I have an app that I'm connecting to a MySQL database. It loses connection in the middle of the night and then spouts about null connections and JDBC hasn't received messages in X seconds.
I call getConnection() before I do anything that requires communication with the SQL server.
This is my getConnection() method:
private Connection getConnection() {
try {
if (connection != null) {
if (connection.isClosed() || !connection.isValid(10000)) {
this.initializeRamsesConnection();
}
} else {
this.initializeRamsesConnection();
}
} catch (Exception e) {
debug("Connection failed: " + e);
}
return connection;
}
In the initializeRamsesConnection() method I put the password and so on information into a string and then I create the connection in the standard JDBC way.
Then I call this method:
private Connection getConnectionFromConnectionString() {
Connection con = null;
String driver = "com.mysql.jdbc.Driver";
try {
Class.forName(driver);//jdbc sorcery
//if there is no connection string
if (getConnectionString() == null) {
HMIDatabaseAdapter.debug("No connection string");
}
//makes a string out of the values of db/host
String str = getConnectionString();
//if there is no driver
if (driver == null) {
debug("" + ": " + "No driver");
}
//Tries to make a connection from the connection string, username, and the password.
con = DriverManager.getConnection(str, username, password);
//if for some reason the connection is null
if (con == null) {
HMIDatabaseAdapter.debug("CONNECTION IS NULL, WHAT?");
}
} catch (Exception ex) {
HMIDatabaseAdapter.debug("getConnection() " + ex);
}
return con;
}
What can I change in either of these methods to accommodate losing connection?
This is not the correct way of retrieving a connection. You're retrieving the connection and assigning it as an instance (or worse, static) variable of the class. Basically, you're keeping the connection open forever and reusing a single connection for all queries. This may end up in a disaster if the queries are executed by different threads. Also, when it's been kept open for too long, the DB will reclaim it because it assumes that it's dead/leaked.
You should acquire and close the connection in the shortest possible scope. I.e. in the very same try block as where you're executing the query. Something like this:
public Entity find(Long id) throws SQLException {
Entity entity = null;
try (
Connection connection = dataSource.getConnection(); // This should return a NEW connection!
PreparedStatement statement = connection.prepareStatement(SQL_FIND);
) {
statement.setLong(1, id);
try (ResultSet resultSet = preparedStatement.executeQuery()) {
if (resultSet.next()) {
entity = new Entity(
resultSet.getLong("id"),
resultSet.getString("name"),
resultSet.getInt("value")
);
}
}
}
return entity;
}
If you worry about connecting performance and want to reuse connections, then you should be using a connection pool. You could homegrow one, but I strongly discourage this as you seem to be pretty new to the stuff. Just use an existing connection pool like BoneCP, C3P0 or DBCP. Note that you should not change the JDBC idiom as shown in the above example. You still need to acquire and close the connection in the shortest possible scope. The connection pool will by itself worry about actually reusing, testing and/or closing the connection.
See also:
Am I Using JDBC Connection Pooling?
JDBC MySql connection pooling practices to avoid exhausted connection pool
Where in your code are the errors on losing connection coming from? This would probably be the best place to start.
Off the top of my head (and I may be wrong), JDBC connections will only close on an actual fatal error, so you won't know they've failed until you try to do something.
What I've done in the past is to invalidate the connection at the point of failure and retry periodically.
Maybe this is what you are looking for:
http://dev.mysql.com/doc/refman/5.0/en/auto-reconnect.html
For java see autoReconnect:
http://dev.mysql.com/doc/refman/5.0/en/connector-j-reference-configuration-properties.html
I am (successfully) connecting to a database using the following:
java.sql.Connection connect = DriverManager.getConnection(
"jdbc:mysql://localhost/some_database?user=some_user&password=some_password");
What should I be checking to see if the connection is still open and up after some time?
I was hoping for something like connect.isConnected(); available for me to use.
Your best chance is to just perform a simple query against one table, e.g.:
select 1 from SOME_TABLE;
Oh, I just saw there is a new method available since 1.6:
java.sql.Connection.isValid(int timeoutSeconds):
Returns true if the connection has not been closed and is still valid.
The driver shall submit a query on the connection or use some other
mechanism that positively verifies the connection is still valid when
this method is called. The query submitted by the driver to validate
the connection shall be executed in the context of the current
transaction.
Nothing. Just execute your query. If the connection has died, either your JDBC driver will reconnect (if it supports it, and you enabled it in your connection string--most don't support it) or else you'll get an exception.
If you check the connection is up, it might fall over before you actually execute your query, so you gain absolutely nothing by checking.
That said, a lot of connection pools validate a connection by doing something like SELECT 1 before handing connections out. But this is nothing more than just executing a query, so you might just as well execute your business query.
Use Connection.isClosed() function.
The JavaDoc states:
Retrieves whether this Connection object has been closed. A
connection is closed if the method close has been called on it or if
certain fatal errors have occurred. This method is guaranteed to
return true only when it is called after the method Connection.close
has been called.
You also can use
public boolean isDbConnected(Connection con) {
try {
return con != null && !con.isClosed();
} catch (SQLException ignored) {}
return false;
}
If you are using MySQL
public static boolean isDbConnected() {
final String CHECK_SQL_QUERY = "SELECT 1";
boolean isConnected = false;
try {
final PreparedStatement statement = db.prepareStatement(CHECK_SQL_QUERY);
isConnected = true;
} catch (SQLException | NullPointerException e) {
// handle SQL error here!
}
return isConnected;
}
I have not tested with other databases. Hope this is helpful.
The low-cost method, regardless of the vendor implementation, would be to select something from the process memory or the server memory, like the DB version or the name of the current database. IsClosed is very poorly implemented.
Example:
java.sql.Connection conn = <connect procedure>;
conn.close();
try {
conn.getMetaData();
} catch (Exception e) {
System.out.println("Connection is closed");
}
Here is a simple solution if you are using JDBC to get the default connection
private Connection getDefaultConnection() throws SQLException, ApiException {
Connection connection = null;
try {
connection = dataSource.getConnection ();
}catch (SQLServerException sqlException) {
// DB_UNAVAILABLE EXCEPTION
}
return connection;
}
I am using JDBC to connect to a DB. Since the network was slow, i could get the connection after some 2 or 3 retry manually. Is it possible to retry the connection automatically if connection fails? I am using SQLServer 2008 database.
Thanks
A bit decent connection pool is already configureable to do so, for example BoneCP. Most do even do it by default. If you don't use a connection pool but just the basic DriverManager#getConnection() approach, then you have to re-execute it yourself in a while loop as long as the Connection is null.
Here's a basic kickoff example:
Connection connection = null;
while (connection == null) {
try {
connection = DriverManager.getConnection(url, username, password);
} catch (SQLException e) {
logger.info("Connecting failed, retrying...");
}
}
return connection;
This can of course be optimized more with bit longer pauses in between and by setting a maximum retry count, etcetera.
Here the code by which it re-try it 3 times to connect,you can also change the cont instead of 3 this can be changed to any number of times(5,10 etc)
Connection connection = null;
while (connection == null && count<3){
try {
String Connection="jdbc:sqlserver://"+serverip+';'+"database="+dbName+';'+"user=" +Username+';'+"password=" +Password;
Class.forName(classname);
connection = DriverManager.getConnection(Connection);
}catch (SQLException e){
count++;
System.err.println("Connecting failed, retrying...");
}
}
count=0;
return connection;
}