I have a Java app that opens a connection to a database at the beginning, and closes it at the end. However, the program doesn't always finish, because an exception is thrown or I am debugging it and stop it halfway through.
Will this cause open connections to pile up and slow the database, or will it be cleaned up automatically?
A database Connection is owned and managed by the database, the class just gives you access to that database resource. If you don't close the connection then the Java class may be garbage collected, but the database may not be able to tell that the connection is no longer in use which may result in database resources being wasted (until a timeout on the database side) or even leak.
So, when you're done with using your Connection, you should be sure to explicitly close it by calling its close() method. This will allow the garbage collector to recollect memory as early as possible and, more important, it releases any other database resources (cursors, handles, etc) the connection may be holding on to.
The traditional way to do this in Java is to close your ResultSet, Statement, and Connection (in that order) in a finally block when you are done with them and the safe pattern looks like that:
Connection conn = null;
PreparedStatement ps = null;
ResultSet rs = null;
try {
// Do stuff
...
} catch (SQLException ex) {
// Exception handling stuff
...
} finally {
if (rs != null) {
try {
rs.close();
} catch (SQLException e) { /* ignored */}
}
if (ps != null) {
try {
ps.close();
} catch (SQLException e) { /* ignored */}
}
if (conn != null) {
try {
conn.close();
} catch (SQLException e) { /* ignored */}
}
}
The finally block can be slightly improved into (to avoid the null check):
} finally {
try { rs.close(); } catch (Exception e) { /* ignored */ }
try { ps.close(); } catch (Exception e) { /* ignored */ }
try { conn.close(); } catch (Exception e) { /* ignored */ }
}
But, still, this is extremely verbose so you generally end up using an helper class to close the objects in null-safe helper methods and the finally block becomes something like that:
} finally {
DbUtil.closeQuietly(rs);
DbUtil.closeQuietly(ps);
DbUtil.closeQuietly(conn);
}
And, actually, the Apache Commons DbUtils has a DbUtils class which is precisely doing that so there is no need to write your own.
In your case, this will solve the problem of the exception, but not the debugging one (and you will waste database resources until the timeout occur on the database side). So 1. don't debug your code using a production database 2. try to execute your debug session until the end.
Here's what Sun (err...Oracle?) says:
It is recommended that programmers explicitly close connections and statements they have created when they are no longer needed.
A programmer writing code in the Java programming language and not using any outside resources does not need to worry about memory management. The garbage collector automatically removes objects when they are no longer being used and frees the memory they were using. When memory is running low, it will recycle discarded objects, making the memory they currently occupy available for quick reuse.
However, if an application uses external resources, as it does when it accesses a DBMS with the JDBC API, the garbage collector has no way of knowing the status of those resources. It will still recycle discarded objects, but if there is lots of free memory in the Java heap, it may garbage collect infrequently, even though the (small) amount of Java garbage is holding open large amounts of expensive database resources. Therefore, it is recommended that programmers explicitly close all connections (with the method Connection.close) and statements (with the method Statement.close) as soon as they are no longer needed, thereby freeing DBMS resources as early as possible. This applies especially to applications that are intended to work with different DBMSs because of variations from one DBMS to another.
I would put the database access in a try block and make sure to close all statements and connections in a finally block.
Your db server will have a timeout setting. It will close the connection and roll back any uncommitted transactions. This has been happening for decades on any production capable db product.
If you want to do it properly use a try { ..your code..} finally { ..close connections..}
Nope.
If your program continues and your connections is alive, then the BD simply rejected your sentence.
If something happened with your connection (by example, a timeout) then the BD was who closed that connection and it's not consuming resources.
If you released your connection, and the garbage collector was called (it can be a while) the connection will close itself before get freed.
If your program terminated without closing your connection, then all the (operating system) process will release its native resources, and between them, the native resource that connected to the BD (probable a network socket). The BD will then receive the connection aborted/closed and release your connection.
The only thing that could happen is that one only execution would connect many times to the BD and do things very bad to keep them open, ocuppying all the connections available. But it's not your case I think.
Edit: in general BD's are made bad-client-behavior-proof
Related
I've build a crawler which is currently experiencing Memory leaks at a speed of 600MB/day. I think the cause is database connection.
Currently, I'm creating a single database connection as a static method as follows:
static
{
try
{
String hostname = PropertyReader.getProperty("hostname");
String port = PropertyReader.getProperty("port");
String dbname = PropertyReader.getProperty("dbname");
String username = PropertyReader.getProperty("username");
String password = PropertyReader.getProperty("password");
Class.forName("com.mysql.jdbc.Driver");
String url = "jdbc:mysql://"+hostname+":"+port+"/"+dbname+"";
conn = DriverManager.getConnection(url, username, password);
System.out.println("conn built");
}
catch (SQLException e)
{
e.printStackTrace();
} catch (ClassNotFoundException e)
{
e.printStackTrace();
}
}
and I'm using this variable in multiple methods as follows:
public static void getSeed()
public static void processPage(String URL)
for retrieving and inserting data into database.
What is the best alternate to avoid memory leaks ?
The biggest source of memory leaks is forgetting to close ResultSets and PreparedStatements. Can't tell if you are doing that elsewhere in your code, but that is where I would start if you do.
Here are my thoughts:
I would not guess about a memory leak. The only reliable way to figure it out is to profile your code. Be a scientist - get data. Use Visual VM with all the plugins installed. It's a great tool and free.
I would not do a database connection this way. Connections are not thread safe. A single connection doesn't feel very scalable, either. Better to have a connection pool that you check in and out in the narrowest scope possible.
You don't show any of the database code, but you need to close all Statement, Connection, and ResultSet instances in a finally block in the scope of the method that creates them. Each one should be closed in an individual try/catch block. This will show up as a problem beyond memory leaks in your crawler: Your database server will eventually run out of cursors. It's a problem both for the client and the database server if you fail to clean up properly.
Different threads should NOT be sharing a single connection. This will lead to grief. Set up a connection pool and then profile. You might be surprised at where the leak shows up.
I have a connection leak in some older Java web applications which do not utilize connection pooling.
Trying to find the leak is hard because IT will not grant me access to v$session SELECT Count(*) FROM v$session;
So instead I am trying to debug with System.out statements. Even after closing the connection conn.close(); when I print conn to the System log file it gives me the connection object name.
try {
Connection conn;
conn.close()
}
catch (SQLException e) { }
finally {
if (conn != null) {
try {
System.out.println("Closing the connection");
conn.close();
}
catch (Exception ex)
{
System.out.println("Exception is " + ex);
}
}
}
// I then check conn and it is not null and I can print the object name.
if (conn != null) {
System.out.println("Connection is still open and is " + conn);
}
however if I also add conn = null; below the conn.close(); statement the connection now seems closed. So my question is does conn.close(); actually release my connection or do I also have to make it null to really release my connection. Like I said it is really hard for me to determine if the connection is actually released without being able to query v$session. Is there snippet of java code which can give me my open connections??
It's probably educational at this point because I plan to refactor these applications to use connection pooling but I'm looking for a quick bandaid for now.
The important part of the close is what's happening on the database side. It's the RDBMS that has to close that connection. Calling the close() method is what communicates the message to the database to close the connection.
Setting the connection to null doesn't instruct RDBMS to do anything.
Same logic applies to ResultSet, which is a cursor on the database side, and Statement. You need to close those in individual try/catch blocks in the finally block of the method that created them, in reverse order of creation. Otherwise you'll see errors about "Max cursors exceeded".
Setting the conn to null only breaks the reference link to the connection object, and has no influence on the connection being open or not. If the connection is still open then the connection will still be referred to from inside the JDBC driver/connection pool etc...
Setting a variable to null is more telling the garbage collector that it is ok to clean up the original object when it wants to than anything else.
As others are saying, you've got two different concepts here: closing the connecting and tracking the connection in a variable.
To close the connection, call conn.close(). This will not set the variable conn to null. You can test if the connection is open with conn.isClosed().
If you don't care to track the connection in your code any more, you can conn = null. This does not immediately close the connection. I believe the connection will be automatically closed, based on the JDBC documentation :
Releases this Connection object's database and JDBC resources immediately instead of waiting for them to be automatically released.
If you choose to go this route, be aware that the garbage collector may not close your connection as quickly as you want, and you may have what appears to be a resource leak; reserved database locks won't be released until the connection is garbage collected. Certain drivers (I don't know if oracle is one) impose maximum limit to the number of connections that may exist at one time, so leaving open connections can also cause failures to connect, later in the program.
Connection leaks are a best. I think a good strategy is to wrap the getting and releasing of connections in a couple of functions and then always get and release your connections through those functions. Then you can have those functions maintain a list of all open connections, and do a stack trace on the caller of the allocate function. Then have a screen that shows a list of all open connections and where they came from. Run this in a test environment, run around using a bunch of screens, then exit them all so all the connections SHOULD close, then bring up the screen that shows open connectoins, and the villain should be revealed.
My explanation here is an educated guess.
As a practice I have always set conn=null after the close. I believe when you do conn.close() you are telling the garbage collector that it's ready to be garbage collected. However, it will be up to the garbage collection process to determine when to do so.
Also you can change your
if(conn!=null)
to
if (conn.isClosed())
..
Is there snippet of Java code which can give me my open connections?
Statement smt = null;
ResultSet rs = null;
try {
// Create Statement from connection
smt = conn.createStatement();
// Execute Query in statement
rs = stmt.executeQuery("SELECT 1 FROM Dual");
if (rs.next()) {
return true; // connection is valid
}
catch (SQLException e) {
// Some sort of logging
return false;
}
finally {
if (smt != null) smt.close();
if (rs != null) rs.close();
}
Just a quick guess, assuming you are using Oracle.
Sugession: Why don't you install jboss and set up connection pooling through there?
I am creating a multi-threaded application. However, I have experienced lots of unexpected behavior from my application when I have one connection object serving all threads.
I am in a dilemma. Should I let every thread create, use and dispose its own connection object or should I use a connection pool?
I have tried connection pooling which makes the application painfully shower. However, my intuition is that if I let every thread create its own connection object, I might get a "too many connection" error.
Please let me know if there is any way to assist in this.
Regards.
Regardless of the threading issue, you should definitely go for a connection pool. It will greatly increase connecting performance. Then to the threading issue, this is indeed a major problem. The normal JDBC idiom is to acquire and close all resources in the shortest possible scope. I.e. all should happen in the very same method block. The problem symptoms which you're describing confirms that you aren't closing those resources properly.
Closing should always happen regardless of whether the connection is coming from a pool or not. Closing a non-pooled connection will prevent it from being timed-out by the database when it's been hold open for a too long time. Closing a pooled connection will actually release it back to the pool and make it available for the next lease.
Here's how the normal JDBC idiom look like for the case of a INSERT.
public void create(Entity entity) throws SQLException {
// Declare.
Connection connection = null;
PreparedStatement statement = null;
try {
// Acquire.
connection = database.getConnection();
statement = connection.prepareStatement(SQL_CREATE);
// Use.
statement.setSomeObject(1, entity.getSomeProperty());
// ...
statement.executeUpdate();
} finally {
// Close.
if (statement != null) try { statement.close(); } catch (SQLException logOrIgnore) {}
if (connection != null) try { connection.close(); } catch (SQLException logOrIgnore) {}
}
}
I'm still an undergrad just working part time and so I'm always trying to be aware of better ways to do things. Recently I had to write a program for work where the main thread of the program would spawn "task" threads (for each db "task" record) which would perform some operations and then update the record to say that it has finished. Therefore I needed a database connection object and PreparedStatement objects in or available to the ThreadedTask objects.
This is roughly what I ended up writing, is creating a PreparedStatement object per thread a waste? I thought static PreparedStatments could create race conditions...
Thread A stmt.setInt();
Thread B stmt.setInt();
Thread A stmt.execute();
Thread B stmt.execute();
A´s version never gets execed..
Is this thread safe? Is creating and destroying PreparedStatement objects that are always the same not a huge waste?
public class ThreadedTask implements runnable {
private final PreparedStatement taskCompleteStmt;
public ThreadedTask() {
//...
taskCompleteStmt = Main.db.prepareStatement(...);
}
public run() {
//...
taskCompleteStmt.executeUpdate();
}
}
public class Main {
public static final db = DriverManager.getConnection(...);
}
I believe it is not a good idea to share database connections (and prepared statements) between threads. JDBC does not require connections to be thread-safe, and I would expect most drivers to not be.
Give every thread its own connection (or synchronize on the connection for every query, but that probably defeats the purpose of having multiple threads).
Is creating and destroying PreparedStatement objects that are always the same not a huge waste?
Not really. Most of the work happens on the server, and will be cached and re-used there if you use the same SQL statement. Some JDBC drivers also support statement caching, so that even the client-side statement handle can be re-used.
You could see substantial improvement by using batched queries instead of (or in addition to) multiple threads, though. Prepare the query once, and run it for a lot of data in a single big batch.
The threadsafety is not the issue here. All looks syntactically and functionally fine and it should work for about half a hour. Leaking of resources is however the real issue here. The application will crash after about half a hour because you never close them after use. The database will in turn sooner or later close the connection itself so that it can claim it back.
That said, you don't need to worry about caching of preparedstatements. The JDBC driver and the DB will take care about this task. Rather worry about resource leaking and make your JDBC code as solid as possible.
public class ThreadedTask implements runnable {
public run() {
Connection connection = null;
Statement statement = null;
try {
connection = DriverManager.getConnection(url);
statement = connection.prepareStatement(sql);
// ...
} catch (SQLException e) {
// Handle?
} finally {
if (statement != null) try { statement.close(); } catch (SQLException logOrIgnore) {}
if (connection != null) try { connection.close(); } catch (SQLException logOrIgnore) {}
}
}
}
To improve connecting performance, make use of a connection pool like c3p0 (this by the way does not mean that you can change the way how you write the JDBC code; always acquire and close the resources in the shortest possible scope in a try-finally block).
You're best to use a connection pool and get each thread to request a connection from the pool. Create your statements on the connection you're handed, remembering to close it and so release it back to the pool when you're done. The benefit of using the pool is that you can easily increase the number of available connections should you find that thread concurrency is becoming an issue.
I've been doing code review (mostly using tools like FindBugs) of one of our pet projects and FindBugs marked following code as erroneous (pseudocode):
Connection conn = dataSource.getConnection();
try{
PreparedStatement stmt = conn.prepareStatement();
//initialize the statement
stmt.execute();
ResultSet rs = stmt.getResultSet();
//get data
}finally{
conn.close();
}
The error was that this code might not release resources. I figured out that the ResultSet and Statement were not closed, so I closed them in finally:
finally{
try{
rs.close()
}catch(SqlException se){
//log it
}
try{
stmt.close();
}catch(SqlException se){
//log it
}
conn.close();
}
But I encountered the above pattern in many projects (from quite a few companies), and no one was closing ResultSets or Statements.
Did you have troubles with ResultSets and Statements not being closed when the Connection is closed?
I found only this and it refers to Oracle having problems with closing ResultSets when closing Connections (we use Oracle db, hence my corrections). java.sql.api says nothing in Connection.close() javadoc.
One problem with ONLY closing the connection and not the result set, is that if your connection management code is using connection pooling, the connection.close() would just put the connection back in the pool. Additionally, some database have a cursor resource on the server that will not be freed properly unless it is explicitly closed.
I've had problems with unclosed ResultSets in Oracle, even though the connection was closed. The error I got was
"ORA-01000: maximum open cursors exceeded"
So: Always close your ResultSet!
You should always close all JDBC resources explicitly. As Aaron and John already said, closing a connection will often only return it to a pool and not all JDBC drivers are implemented exact the same way.
Here is a utility method that can be used from a finally block:
public static void closeEverything(ResultSet rs, Statement stmt,
Connection con) {
if (rs != null) {
try {
rs.close();
} catch (SQLException e) {
}
}
if (stmt != null) {
try {
stmt.close();
} catch (SQLException e) {
}
}
if (con != null) {
try {
con.close();
} catch (SQLException e) {
}
}
}
Oracle will give you errors about open cursors in this case.
According to: http://java.sun.com/javase/6/docs/api/java/sql/Statement.html
it looks like reusing a statement will close any open resultsets, and closing a statement will close any resultsets, but i don't see anything about closing a connection will close any of the resources it created.
All of those details are left to the JDBC driver provider.
Its always safest to close everything explicitly. We wrote a util class that wraps everything with try{ xxx } catch (Throwable {} so that you can just call Utils.close(rs) and Utils.close(stmt), etc without having to worry about exceptions that close scan supposedly throw.
The ODBC Bridge can produce a memory leak with some ODBC drivers.
If you use a good JDBC driver then you should does not have any problems with closing the connection. But there are 2 problems:
Does you know if you have a good driver?
Will you use other JDBC drivers in the future?
That the best practice is to close it all.
I work in a large J2EE web environment. We have several databases that may be connected to in a single request. We began getting logical deadlocks in some of our applications. The issue was that as follows:
User would request page
Server connects to DB 1
Server Selects on DB 1
Server "closes" connection to DB 1
Server connects to DB 2
Deadlocked!
This occurred for 2 reasons, we were experiencing far higher volume of traffic than normal and the J2EE Spec by default does not actually close your connection until the thread finishes execution. So, in the above example step 4 never actually closed the connection even though they were closed properly in finally .
To fix this, you you have to use resource references in the web.xml for your Database Connections and you have to set the res-sharing-scope to unsharable.
Example:
<resource-ref>
<description>My Database</description>
<res-ref-name>jdbc/jndi/pathtodatasource</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
<res-sharing-scope>Unshareable</res-sharing-scope>
</resource-ref>
I've definitely seen problems with unclosed ResultSets, and what can it hurt to close them all the time, right? The unreliability of needing to remembering to do this is one of the best reasons to move to frameworks that manage these details for you. It might not be feasible in your development environment, but I've had great luck using Spring to manage JPA transactions. The messy details of opening connections, statements, result sets, and writing over-complicated try/catch/finally blocks (with try/catch blocks in the finally block!) to close them again just disappears, leaving you to actually get some work done. I'd highly recommend migrating to that kind of a solution.
In Java, Statements (not Resultsets) correlate to Cursors in Oracle. It is best to close the resources that you open as unexpected behavior can occur in regards to the JVM and system resources.
Additionally, some JDBC pooling frameworks pool Statements and Connections, so not closing them might not mark those objects as free in the pool, and cause performance issues in the framework.
In general, if there is a close() or destroy() method on an object, there's a reason to call it, and to ignore it is done so at your own peril.