Memory leaks in Java: database connection in multiple methods - java

I've build a crawler which is currently experiencing Memory leaks at a speed of 600MB/day. I think the cause is database connection.
Currently, I'm creating a single database connection as a static method as follows:
static
{
try
{
String hostname = PropertyReader.getProperty("hostname");
String port = PropertyReader.getProperty("port");
String dbname = PropertyReader.getProperty("dbname");
String username = PropertyReader.getProperty("username");
String password = PropertyReader.getProperty("password");
Class.forName("com.mysql.jdbc.Driver");
String url = "jdbc:mysql://"+hostname+":"+port+"/"+dbname+"";
conn = DriverManager.getConnection(url, username, password);
System.out.println("conn built");
}
catch (SQLException e)
{
e.printStackTrace();
} catch (ClassNotFoundException e)
{
e.printStackTrace();
}
}
and I'm using this variable in multiple methods as follows:
public static void getSeed()
public static void processPage(String URL)
for retrieving and inserting data into database.
What is the best alternate to avoid memory leaks ?

The biggest source of memory leaks is forgetting to close ResultSets and PreparedStatements. Can't tell if you are doing that elsewhere in your code, but that is where I would start if you do.

Here are my thoughts:
I would not guess about a memory leak. The only reliable way to figure it out is to profile your code. Be a scientist - get data. Use Visual VM with all the plugins installed. It's a great tool and free.
I would not do a database connection this way. Connections are not thread safe. A single connection doesn't feel very scalable, either. Better to have a connection pool that you check in and out in the narrowest scope possible.
You don't show any of the database code, but you need to close all Statement, Connection, and ResultSet instances in a finally block in the scope of the method that creates them. Each one should be closed in an individual try/catch block. This will show up as a problem beyond memory leaks in your crawler: Your database server will eventually run out of cursors. It's a problem both for the client and the database server if you fail to clean up properly.
Different threads should NOT be sharing a single connection. This will lead to grief. Set up a connection pool and then profile. You might be surprised at where the leak shows up.

Related

if I forget to close the connection to the database

if i do close to connection database instead if i do close my resultSet ,what happend ?
I did write example to under.
For example.
Database.java
public class Database{
try{
Connection con;
PreparedStatement statement;
ResultSet resultSet;
public static void main{
Class.forName("com.mysql.jdbc.Driver");
String url = "jdbc:mysql://localhost:3306/database";
String _username="root";
String _password = "password";
Connection db = DriverManager.getConnection(url,_username,_password);
String sqlQuery = "select * from category";
statement = db.prepareStatement(sqlQuery);
resultSet = statement .executeQuery();
while(resuletSet.next()){
......
}
// i dont close connect.
resultSet.close();
//what happen then it? connection.close() , will it be automatically down ?
}catch(Exception e){
e.printStackTrace();
}
}
}
You're right to close your ResultSet carefully. Leaving those objects around after you finish with them is definitely a way to get memory leaks.
You should also close your Connection objects when you no longer need them. If your Java program terminates any Connection objects you still have open are closed, automatically. If no queries are in progress on those objects then the MySQL server closes the connection and cleans up when Java abandons the connection.
Java since version 7 has a way to manage this cleanly. You can do
try (Connection db = DriverManager.getConnection(url,_username,_password)){
//use the connection object
...
try (ResultSet result = stmt.ExecuteQuery(whatever)) {
// use the resultset
} catch (whatever) { whatever }
}
catch(whatever) { whatever }
This is is a nice way to avoid these leaks; the Connection and ResultSet objects get closed automatically at the end of the try / catch block just as if they were closed in a finally{} clause. It's called automatic resource block management.
Yes, close Connections when you're done with them. If you open a lot of Connection objects without closing them, two bad things happen:
Less bad: your Java program's RAM will leak.
More bad: your MySQL server's connection slots will fill up and it will start rejecting new connections. Client bugs which use up server resources are generally bad bugs.
Failing to close Connections can be pernicious, because typical programs don't use as many of them as ResultSets. So it takes longer to accumulate lots of unclosed connections, and you may not detect the problem while testing. Testers should log into the MySQL server directly and run the SHOW PROCESSLIST; command during system testing to see if unclosed Connections are accumulating.
Close your ResultSets and your Connections.

Java SQLite multiple connections in the same class tabbedpane [duplicate]

I have a bukkit plugin (minecraft) that requires a connection to the database.
Should a database connection stay open all the time, or be opened and closed when needed?
The database connection must be opened only when its needed and closed after doing all the necessary job with it. Code sample:
Prior to Java 7:
Connection con = null;
try {
con = ... //retrieve the database connection
//do your work...
} catch (SQLException e) {
//handle the exception
} finally {
try {
if (con != null) {
con.close();
}
} catch (SQLException shouldNotHandleMe) {
//...
}
}
Java 7:
try (Connection con = ...) {
} catch (SQLException e) {
}
//no need to call Connection#close since now Connection interface extends Autocloseable
But since manually opening a database connection is too expensive, it is highly recommended to use a database connection pool, represented in Java with DataSource interface. This will handle the physical database connections for you and when you close it (i.e. calling Connection#close), the physical database connection will just be in SLEEP mode and still be open.
Related Q/A:
Java Connection Pooling
Some tools to handle database connection pooling:
BoneCP
c3po
Apache Commons DBCP
HikariCP
Depends on what are your needs.
Creating a connection takes some time, so if you need to access database frequently it's better to keep the connection open. Also it's better to create a pool, so that many users can access database simultaneously(if it's needed).
If you need to use this connection only few times you may not keep it open, but you will have delay when you would like to access database. So i suggest you to make a timer that will keep connection open for some time(connection timeout).
You need to close your connections after each query executions.Sometimes you need to execute multiple queries at the same time because the queries are hanging from each other.Such as "first insert task then assign it to the employees".At this time execute your queries on the same transaction and commit it, if some errors occur then rollback.By default autocommit is disabled in JDBC. Example
Use connection pooling.If you are developing a webapplication then use App Server connection pooling.App server will use the same pooling for each of your applications so you can control the connection count from the one point.Highly recommend the Apache Tomcat Connection pooling.Example
As an additional info:
Connection, Statement and ResultSet.
1.If you close connection you don't need close statement or resultset.Both of them will be closed automatically
2.If you close Statement it will close ResultSet also
3.if you use try-with-resources like this:
try (Connection con = ...) {
} catch (SQLException e) {
}
it will close the connection automatically.Because try-with-resources require autoclosable objects and Connection is autocloseable.You can see the details about try-with-resources here
Actually, it's all matter on how you write your application! It's an art, but sadly everyone takes a tutorial for a good practice like Microsoft's tutorials.
If you know what you are coding, then you keep your connection open for the lifetime of the application. It's simple, not because you have to go at work in the morning that everyday we have to build a special route just for you! You take that single route or 2 or 4 like everyone does! You judge for the traffics and you build 2, 4 or 6 routes as needed. If there is traffic with these 4 or 6 routes, you wait!
Happy coding.
The Connection should be opened only when required. If it is open before the actual need, it reduces one active connection from the connection pool..so it ultimately effects the users of the application.
So,it is always a better practice to open connection only when required and closing it after completion of process.
Always try puttting you connection close logic inside the finally block that will ensure that your connection will be closed,even if any exception occurs in the application
finally
{
connection.close()
}

Sharing database connection in Java EE

I am developing a web application using JSP & Servlet (IDE: Eclipse, Database: Oracle10).
I have developed java class which returns a static connection, and that connection will be used by my entire web application.
public class DBConnection
{
private static Connection con = null;
static Connection getConnection(String str)
{
try
{
Class.forName("oracle.jdbc.driver.OracleDriver");
con = DriverManager.getConnection("MyValuesHere");
System.out.println("New instance of Connection is created for: "+str);
}
catch(ClassNotFoundException cnfe)
{
System.out.println("Error loading class!");
cnfe.printStackTrace();
}
catch(SQLException sqle)
{
System.out.println("Error connecting to the database!");
sqle.printStackTrace();
}
return con;
}//getConnection()
}//class
Above class is working fine. Then I have another 4 java classes for
Inserting
Updating
Deleting
Selecting
data from database using the above connection. So in those 4 classes I am getting connection which is created in my DBConnection class, and those four classes are also working fine. This four classes are used in my all Servlet's.
To get Connection in those 4 classes I am writing following line:
private static Connection con = DBConnection.getConnection("UpdateQuery.java");
But problem is that I want to share the same connection in all four classes, but connection is created separately in those 4 classes. So how should I share the same connection in those four classes? is there better way of doing this? and if I use this approach will there be any issues in web application because of sharing the connection for whole application?
You are (implicitly) trying to solve a non-trivial task.
Such things are normally done by the container - taking connections from a pool, then returning them back, reconnection etc...
If you use a fully functional applications server you'd better configure and use data sources.
If your server doesn't support data sources, do not mess up with saving connection into a private field. What for example happenes when your connection is lost? Your private variable will have a non-working connection. Do you have any recovery mechanism?
Your code will be much more robust if you get it in the beginning of the business operation and then close it.
Or try to find a professionally written library that supports connection pools - it will do pretty much the same as a classic container in handling a connection pool.
Or write it yourself, but it will be a separate task with many questions.
Looks like you wanted to turn Connection into a singleton but then forgot to check whether it's been instantiated already. In getConnection you could check if con is not null in the first place and return that instance right away. Only if con is still null, proceed with initialization.
You should save the created connection instance into a private static field in DBConnection, and when getConnection is called, you check if the field is null, then create the connection, then return it:
if (connection == null) {
connection = createConnection();
}
return connection;
where connection is a private static Connection connection field of DBConnection class.
However I strongly suggest to not use this approach as sharing a connection between concurrent threads will cause serious problems. I suggest to use connection pooling

Oracle connection not closing in Java Application

I have a connection leak in some older Java web applications which do not utilize connection pooling.
Trying to find the leak is hard because IT will not grant me access to v$session SELECT Count(*) FROM v$session;
So instead I am trying to debug with System.out statements. Even after closing the connection conn.close(); when I print conn to the System log file it gives me the connection object name.
try {
Connection conn;
conn.close()
}
catch (SQLException e) { }
finally {
if (conn != null) {
try {
System.out.println("Closing the connection");
conn.close();
}
catch (Exception ex)
{
System.out.println("Exception is " + ex);
}
}
}
// I then check conn and it is not null and I can print the object name.
if (conn != null) {
System.out.println("Connection is still open and is " + conn);
}
however if I also add conn = null; below the conn.close(); statement the connection now seems closed. So my question is does conn.close(); actually release my connection or do I also have to make it null to really release my connection. Like I said it is really hard for me to determine if the connection is actually released without being able to query v$session. Is there snippet of java code which can give me my open connections??
It's probably educational at this point because I plan to refactor these applications to use connection pooling but I'm looking for a quick bandaid for now.
The important part of the close is what's happening on the database side. It's the RDBMS that has to close that connection. Calling the close() method is what communicates the message to the database to close the connection.
Setting the connection to null doesn't instruct RDBMS to do anything.
Same logic applies to ResultSet, which is a cursor on the database side, and Statement. You need to close those in individual try/catch blocks in the finally block of the method that created them, in reverse order of creation. Otherwise you'll see errors about "Max cursors exceeded".
Setting the conn to null only breaks the reference link to the connection object, and has no influence on the connection being open or not. If the connection is still open then the connection will still be referred to from inside the JDBC driver/connection pool etc...
Setting a variable to null is more telling the garbage collector that it is ok to clean up the original object when it wants to than anything else.
As others are saying, you've got two different concepts here: closing the connecting and tracking the connection in a variable.
To close the connection, call conn.close(). This will not set the variable conn to null. You can test if the connection is open with conn.isClosed().
If you don't care to track the connection in your code any more, you can conn = null. This does not immediately close the connection. I believe the connection will be automatically closed, based on the JDBC documentation :
Releases this Connection object's database and JDBC resources immediately instead of waiting for them to be automatically released.
If you choose to go this route, be aware that the garbage collector may not close your connection as quickly as you want, and you may have what appears to be a resource leak; reserved database locks won't be released until the connection is garbage collected. Certain drivers (I don't know if oracle is one) impose maximum limit to the number of connections that may exist at one time, so leaving open connections can also cause failures to connect, later in the program.
Connection leaks are a best. I think a good strategy is to wrap the getting and releasing of connections in a couple of functions and then always get and release your connections through those functions. Then you can have those functions maintain a list of all open connections, and do a stack trace on the caller of the allocate function. Then have a screen that shows a list of all open connections and where they came from. Run this in a test environment, run around using a bunch of screens, then exit them all so all the connections SHOULD close, then bring up the screen that shows open connectoins, and the villain should be revealed.
My explanation here is an educated guess.
As a practice I have always set conn=null after the close. I believe when you do conn.close() you are telling the garbage collector that it's ready to be garbage collected. However, it will be up to the garbage collection process to determine when to do so.
Also you can change your
if(conn!=null)
to
if (conn.isClosed())
..
Is there snippet of Java code which can give me my open connections?
Statement smt = null;
ResultSet rs = null;
try {
// Create Statement from connection
smt = conn.createStatement();
// Execute Query in statement
rs = stmt.executeQuery("SELECT 1 FROM Dual");
if (rs.next()) {
return true; // connection is valid
}
catch (SQLException e) {
// Some sort of logging
return false;
}
finally {
if (smt != null) smt.close();
if (rs != null) rs.close();
}
Just a quick guess, assuming you are using Oracle.
Sugession: Why don't you install jboss and set up connection pooling through there?

Problem with not closing db connection while debugging?

I have a Java app that opens a connection to a database at the beginning, and closes it at the end. However, the program doesn't always finish, because an exception is thrown or I am debugging it and stop it halfway through.
Will this cause open connections to pile up and slow the database, or will it be cleaned up automatically?
A database Connection is owned and managed by the database, the class just gives you access to that database resource. If you don't close the connection then the Java class may be garbage collected, but the database may not be able to tell that the connection is no longer in use which may result in database resources being wasted (until a timeout on the database side) or even leak.
So, when you're done with using your Connection, you should be sure to explicitly close it by calling its close() method. This will allow the garbage collector to recollect memory as early as possible and, more important, it releases any other database resources (cursors, handles, etc) the connection may be holding on to.
The traditional way to do this in Java is to close your ResultSet, Statement, and Connection (in that order) in a finally block when you are done with them and the safe pattern looks like that:
Connection conn = null;
PreparedStatement ps = null;
ResultSet rs = null;
try {
// Do stuff
...
} catch (SQLException ex) {
// Exception handling stuff
...
} finally {
if (rs != null) {
try {
rs.close();
} catch (SQLException e) { /* ignored */}
}
if (ps != null) {
try {
ps.close();
} catch (SQLException e) { /* ignored */}
}
if (conn != null) {
try {
conn.close();
} catch (SQLException e) { /* ignored */}
}
}
The finally block can be slightly improved into (to avoid the null check):
} finally {
try { rs.close(); } catch (Exception e) { /* ignored */ }
try { ps.close(); } catch (Exception e) { /* ignored */ }
try { conn.close(); } catch (Exception e) { /* ignored */ }
}
But, still, this is extremely verbose so you generally end up using an helper class to close the objects in null-safe helper methods and the finally block becomes something like that:
} finally {
DbUtil.closeQuietly(rs);
DbUtil.closeQuietly(ps);
DbUtil.closeQuietly(conn);
}
And, actually, the Apache Commons DbUtils has a DbUtils class which is precisely doing that so there is no need to write your own.
In your case, this will solve the problem of the exception, but not the debugging one (and you will waste database resources until the timeout occur on the database side). So 1. don't debug your code using a production database 2. try to execute your debug session until the end.
Here's what Sun (err...Oracle?) says:
It is recommended that programmers explicitly close connections and statements they have created when they are no longer needed.
A programmer writing code in the Java programming language and not using any outside resources does not need to worry about memory management. The garbage collector automatically removes objects when they are no longer being used and frees the memory they were using. When memory is running low, it will recycle discarded objects, making the memory they currently occupy available for quick reuse.
However, if an application uses external resources, as it does when it accesses a DBMS with the JDBC API, the garbage collector has no way of knowing the status of those resources. It will still recycle discarded objects, but if there is lots of free memory in the Java heap, it may garbage collect infrequently, even though the (small) amount of Java garbage is holding open large amounts of expensive database resources. Therefore, it is recommended that programmers explicitly close all connections (with the method Connection.close) and statements (with the method Statement.close) as soon as they are no longer needed, thereby freeing DBMS resources as early as possible. This applies especially to applications that are intended to work with different DBMSs because of variations from one DBMS to another.
I would put the database access in a try block and make sure to close all statements and connections in a finally block.
Your db server will have a timeout setting. It will close the connection and roll back any uncommitted transactions. This has been happening for decades on any production capable db product.
If you want to do it properly use a try { ..your code..} finally { ..close connections..}
Nope.
If your program continues and your connections is alive, then the BD simply rejected your sentence.
If something happened with your connection (by example, a timeout) then the BD was who closed that connection and it's not consuming resources.
If you released your connection, and the garbage collector was called (it can be a while) the connection will close itself before get freed.
If your program terminated without closing your connection, then all the (operating system) process will release its native resources, and between them, the native resource that connected to the BD (probable a network socket). The BD will then receive the connection aborted/closed and release your connection.
The only thing that could happen is that one only execution would connect many times to the BD and do things very bad to keep them open, ocuppying all the connections available. But it's not your case I think.
Edit: in general BD's are made bad-client-behavior-proof

Categories

Resources