Develop Reference

Java SQL memory leak

I'm facing an issue where I have a java application running on a server, and it starts growing in memory until eventually the server cannot handle it anymore.
This is some sort of memory leak / resource leak problem, which I thought was extremely rare in Java due to the garbage collection. I guess something is being referenced and never used, so the garbage collector does not collect it.
The problem is that the size in memory grows so slowly that I'm not able to debug it properly (it may take two weeks to make the server unusable).
I'm using java + mysql-connector, and I'm sure the memory leak is caused by something related to the database connection.
Here is how I connect to the database:
private static Connection connect(){
try {
Connection conn = null;
conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/database","client","password");
return conn;
}catch(SQLException ex){
System.out.println("SQLException: " + ex.getMessage());
System.out.println("SQLState: " + ex.getSQLState());
System.out.println("VendorError: " + ex.getErrorCode());
return null;
}
}
public static Connection getConnection(){
try {
if (connection == null || connection.isClosed()) connection = connect();
return connection;
}catch (SQLException exception){
System.out.println("exception trying to connect to the database");
return null;
}
}
I can't find any possible problem here, but who knows!
Here's how I retrieve information from the database:
public void addPoints(long userId,int cantidad){
try {
if(DatabaseConnector.getConnection()!=null) {
PreparedStatement stm = DatabaseConnector.getConnection().prepareStatement("UPDATE users SET points = points + ? WHERE id = ? ");
stm.setLong(2, userId);
stm.setInt(1, cantidad);
if(stm.executeUpdate()==0){ //user doesn't have any point records in the database yet
PreparedStatement stm2 = DatabaseConnector.getConnection().prepareStatement("INSERT INTO users (id,points) VALUES (?,?)");
stm2.setLong(1, userId);
stm2.setInt(2, cantidad);
stm2.executeUpdate();
}
}
}catch (SQLException exception){
System.out.println("error recording points");
}
}
public ArrayList<CustomCommand> getCommands(long chatId) throws SQLException{
ArrayList<CustomCommand> commands = new ArrayList<>();
if(DatabaseConnector.getConnection() != null) {
PreparedStatement stm = DatabaseConnector.getConnection().prepareStatement("SELECT text,fileID,commandText,type,probability FROM customcommands WHERE chatid = ?");
stm.setLong(1, chatId);
ResultSet results = stm.executeQuery();
if(!results.isBeforeFirst()) return null;
while (results.next()){
commands.add(new CustomCommand(results.getString(1),results.getString(2),results.getString(3), CustomCommand.Type.valueOf(results.getString(4)),results.getInt(5)));
}
return commands;
}
return null;
}
Maybe the problem is something related to exception catching and statements not being correctly executed? Maybe something related to result sets?
It's driving me crazy. Thanks for helping me!

You do nothing to clean up ResultSet and Statement before you return. That's a bad idea. You should be closing each one in individual try/catch blocks in a finally block.
A ResultSet is an object that represents a database cursor on the database. You should close it so you don't run out of cursors.
I wouldn't have a single static Connection. I'd expect a thread-safe, managed pool of connections.
I wouldn't return a null. You don't make clear what the user is supposed to do with that. Better to throw an exception.

JDBC Pool - How to properly use

I use JDBC pool for my Tomcat web applications.
As I read in docs, I need get connection, execute query and close.
After close, connection returns to pool.
I think Way 1 better when query DB depend on external event and for the task which running every 5sec Way 2 more correct.
Could someone explain which way use for task which repeat every 5 sec?
PS: I skip extra checks in code to make code looks readable.
Way #1 Gets connection from pool and close every 5 sec
Connection c = null;
Statement s = null;
ResultSet rs = null;
DataSource ds = ... Get DataSource ...
while(running) {
try {
c = ds.getConnection();
s = c.createStatement();
rs = s.executeQuery('SELECT data FROM my_table');
... do something with result ...
} catch (SQLException sec) {
... print exception ...
} finally {
try {
rs.close();
s.close();
c.close();
} catch (SQLException sec) { ... print exception ... }
... Thread sleep 5 seconds and repeat ...
}
}
Way #2 Get connection before loop and close after, reconnect inside loop
Connection c = null;
Statement s = null;
ResultSet rs = null;
DataSource ds = ... Get DataSource ...
c = ds.getConnection();
while(running) {
try {
s = c.createStatement();
rs = s.executeQuery('SELECT data FROM my_table');
... do something with result ...
} catch (SQLException sec) {
... print exception ...
... if connection lost, try reconnect and execute query again ...
} finally {
try {
rs.close();
s.close();
} catch (SQLException sec) {
... print exception ...
}
... Thread sleep 5 seconds and repeat ...
}
}
c.close();
Pool config
<Resource name="jdbc/pg_mega" auth="Container"
type="javax.sql.DataSource" driverClassName="org.postgresql.Driver"
url="jdbc:postgresql://127.0.0.1:6432/db"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
username="***" password="****"
defaultAutoCommit="true"
initialSize="1"
maxActive="300"
maxTotal="300"
maxIdle="20"
minIdle="5"
maxWait="10000"
validationQuery="select 1"
validationInterval="30000"
testWhileIdle="false"
testOnBorrow="true"
testOnReturn="false"
timeBetweenEvictionRunsMillis="30000"
minEvictableIdleTimeMillis="30000"
/>

I think the most common pattern is this:
Connection conn = null;
PreparedStatement stmt = null;
ResultSet res = null;
try {
conn = ds.getConnection();
stmt = conn.prepareStatement(sqlStatement);
//
// ....
res = stmt.executeQuery();
// use the resultset
conn.commit();
} catch (SQLException e) {
// Manage the exception
try {
conn.rollback();
} catch (SQLException e1) {
// SWALLOW
}
} finally {
close(res);
close(stmt);
close(conn);
}
I use these helper functions to safely close without too much boilerplate, from java 7 on you can autoclose so these helper are no use anymore.
public static void close(Connection conn) {
try {
if (conn != null)
conn.close();
} catch (SQLException e) {
// SWALLOW
}
}
public static void close(Statement stmt) {
try {
if (stmt != null)
stmt.close();
} catch (SQLException e) {
// SWALLOW
}
}
public static void close(ResultSet res) {
try {
if (res != null)
res.close();
} catch (SQLException e) {
// SWALLOW
}
}
You should really be sure to close the connection in the finally statement, bad things happens if you do not close the connection, and if (for example) the rs is null in your example (not so difficult) you won't close the connection.
Getting and releasing the connection from the pool is not a performance problem, it happens in microseconds , thousands of times faster then any possible query.
The reason why you do not release the connection eagerly is transactions, you want to keep the same connection for the whole transaction (no way around this).
When you commit (or rollback) then you don't need that peculiar connection anymore, so just release it.
Another hint, close the connection in finally, even if you catch SQL Exceptions, because there are always Runtime Exceptions and even Errors (which you will not catch), but finally will be executed even in the face of OutOfMemoryError or ClassDefNotFoundError or any other, and the connection will be returned to the pool.
Last but not least is the pool that will try to reconnect in case of disconnection, in fact the pool will just ditch invalid connection and create a fresh batch when needed.
You should choose a good policy of connection validation, a bad choice will lead to much extra time in getting the connection thus hitting hard on performance, or to exceptions caused by invalid connection acquired from the pool.
Optimizing the pool is like many other performance tuning tasks: HARD .
For example:
testOnBorrow="true" will hit the DB before acquiring the connection, it is safe, but it will cost, tens or hundreds of time slower that not checking on borrow.
testWhileIdle="true" instead is less safe (you could get an invalid connection) but is much faster an has the advantage of keeping alive the connections.
You have to choose considering how you use connection, how you deal with errors, where is the DB (on the same machine, on a lan, on a wan) and many other factors.

Way #2 is not correct when you use a pool. If you use a pool, you should always try to keep the connection out of the pool ("leased") for as short as possible to get the most out of pool usage. If you do not use a pool, you have to consider the cost of creating and destroying connections and manage the connection's life cycle.
If you use a pool, a leased connection must always be returned to the pool (a connection is returned to the pool when you close the connection). If connections are not returned to the pool (i.e. connections are leaked), the pool will be empty soon and your application will stop working. This is especially important when things go wrong (e.g. in your code example, when rs is null due to a query error, the connection will be leaked). To prevent connections from leaking, consider using a tool like Sql2o which has built-in protection against connection leakage.
Also reduce the number of connections in the pool. Start with minIndle="1" and maxActive="4". Use stress-testing to determine the upper-limit of the pool-size (more connections in a pool usually do more harm than good, see also About Pool Sizing from HikariCP which has more good articles about database connection pools).

Is a Mysql connection thread safe

public void SQLconnect() {
try {
System.out.println("Connecting to MySQL database...");
Class.forName("com.mysql.jdbc.Driver").newInstance();
String conn = "jdbc:mysql://" + this.SQL_HOST/* + ":" + this.SQL_PORT */
+ "/" + this.SQL_DATA;
this.con = DriverManager
.getConnection(conn, this.SQL_USER, this.SQL_PASS);
} catch (ClassNotFoundException ex) {
System.err.println("No MySQL driver found!");
} catch (SQLException ex) {
System.err
.println("Error while fetching MySQL connection!");
} catch (Exception ex) {
System.err
.println("Unknown error while fetching MySQL connection.");
}
}
This is java, Can I use the connection "con" to connect to my MySql database from different threads?
Or is this not thread safe.
And if its not thread safe, How should I do this?

It is not thread safe. You have to get a new connection every time you need one. It should never be a member variable, always a local variable or method parameter.

In JDBC, the Connection interface is not thread safe, you have to manage it your self. For example, you need to open new connection every time and close it.
For convenience, you can use Connection Pool, because manage connection's open and close is a boring task; with connection pool, you just get an connection and put it back, the pool will manage all connections it opened and reuse them. You can checkout C3P0 or DBCP

Java JDBC connection status

I am (successfully) connecting to a database using the following:
java.sql.Connection connect = DriverManager.getConnection(
"jdbc:mysql://localhost/some_database?user=some_user&password=some_password");
What should I be checking to see if the connection is still open and up after some time?
I was hoping for something like connect.isConnected(); available for me to use.

Your best chance is to just perform a simple query against one table, e.g.:
select 1 from SOME_TABLE;
Oh, I just saw there is a new method available since 1.6:
java.sql.Connection.isValid(int timeoutSeconds):
Returns true if the connection has not been closed and is still valid.
The driver shall submit a query on the connection or use some other
mechanism that positively verifies the connection is still valid when
this method is called. The query submitted by the driver to validate
the connection shall be executed in the context of the current
transaction.

Nothing. Just execute your query. If the connection has died, either your JDBC driver will reconnect (if it supports it, and you enabled it in your connection string--most don't support it) or else you'll get an exception.
If you check the connection is up, it might fall over before you actually execute your query, so you gain absolutely nothing by checking.
That said, a lot of connection pools validate a connection by doing something like SELECT 1 before handing connections out. But this is nothing more than just executing a query, so you might just as well execute your business query.

Use Connection.isClosed() function.
The JavaDoc states:
Retrieves whether this Connection object has been closed. A
connection is closed if the method close has been called on it or if
certain fatal errors have occurred. This method is guaranteed to
return true only when it is called after the method Connection.close
has been called.

You also can use
public boolean isDbConnected(Connection con) {
try {
return con != null && !con.isClosed();
} catch (SQLException ignored) {}
return false;
}

If you are using MySQL
public static boolean isDbConnected() {
final String CHECK_SQL_QUERY = "SELECT 1";
boolean isConnected = false;
try {
final PreparedStatement statement = db.prepareStatement(CHECK_SQL_QUERY);
isConnected = true;
} catch (SQLException | NullPointerException e) {
// handle SQL error here!
}
return isConnected;
}
I have not tested with other databases. Hope this is helpful.

The low-cost method, regardless of the vendor implementation, would be to select something from the process memory or the server memory, like the DB version or the name of the current database. IsClosed is very poorly implemented.
Example:
java.sql.Connection conn = <connect procedure>;
conn.close();
try {
conn.getMetaData();
} catch (Exception e) {
System.out.println("Connection is closed");
}

Here is a simple solution if you are using JDBC to get the default connection
private Connection getDefaultConnection() throws SQLException, ApiException {
Connection connection = null;
try {
connection = dataSource.getConnection ();
}catch (SQLServerException sqlException) {
// DB_UNAVAILABLE EXCEPTION
}
return connection;
}

Connection Retry using JDBC

I am using JDBC to connect to a DB. Since the network was slow, i could get the connection after some 2 or 3 retry manually. Is it possible to retry the connection automatically if connection fails? I am using SQLServer 2008 database.
Thanks

A bit decent connection pool is already configureable to do so, for example BoneCP. Most do even do it by default. If you don't use a connection pool but just the basic DriverManager#getConnection() approach, then you have to re-execute it yourself in a while loop as long as the Connection is null.
Here's a basic kickoff example:
Connection connection = null;
while (connection == null) {
try {
connection = DriverManager.getConnection(url, username, password);
} catch (SQLException e) {
logger.info("Connecting failed, retrying...");
}
}
return connection;
This can of course be optimized more with bit longer pauses in between and by setting a maximum retry count, etcetera.

Here the code by which it re-try it 3 times to connect,you can also change the cont instead of 3 this can be changed to any number of times(5,10 etc)
Connection connection = null;
while (connection == null && count<3){
try {
String Connection="jdbc:sqlserver://"+serverip+';'+"database="+dbName+';'+"user=" +Username+';'+"password=" +Password;
Class.forName(classname);
connection = DriverManager.getConnection(Connection);
}catch (SQLException e){
count++;
System.err.println("Connecting failed, retrying...");
}
}
count=0;
return connection;
}