Develop Reference

Java SQL memory leak

I'm facing an issue where I have a java application running on a server, and it starts growing in memory until eventually the server cannot handle it anymore.
This is some sort of memory leak / resource leak problem, which I thought was extremely rare in Java due to the garbage collection. I guess something is being referenced and never used, so the garbage collector does not collect it.
The problem is that the size in memory grows so slowly that I'm not able to debug it properly (it may take two weeks to make the server unusable).
I'm using java + mysql-connector, and I'm sure the memory leak is caused by something related to the database connection.
Here is how I connect to the database:
private static Connection connect(){
try {
Connection conn = null;
conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/database","client","password");
return conn;
}catch(SQLException ex){
System.out.println("SQLException: " + ex.getMessage());
System.out.println("SQLState: " + ex.getSQLState());
System.out.println("VendorError: " + ex.getErrorCode());
return null;
}
}
public static Connection getConnection(){
try {
if (connection == null || connection.isClosed()) connection = connect();
return connection;
}catch (SQLException exception){
System.out.println("exception trying to connect to the database");
return null;
}
}
I can't find any possible problem here, but who knows!
Here's how I retrieve information from the database:
public void addPoints(long userId,int cantidad){
try {
if(DatabaseConnector.getConnection()!=null) {
PreparedStatement stm = DatabaseConnector.getConnection().prepareStatement("UPDATE users SET points = points + ? WHERE id = ? ");
stm.setLong(2, userId);
stm.setInt(1, cantidad);
if(stm.executeUpdate()==0){ //user doesn't have any point records in the database yet
PreparedStatement stm2 = DatabaseConnector.getConnection().prepareStatement("INSERT INTO users (id,points) VALUES (?,?)");
stm2.setLong(1, userId);
stm2.setInt(2, cantidad);
stm2.executeUpdate();
}
}
}catch (SQLException exception){
System.out.println("error recording points");
}
}
public ArrayList<CustomCommand> getCommands(long chatId) throws SQLException{
ArrayList<CustomCommand> commands = new ArrayList<>();
if(DatabaseConnector.getConnection() != null) {
PreparedStatement stm = DatabaseConnector.getConnection().prepareStatement("SELECT text,fileID,commandText,type,probability FROM customcommands WHERE chatid = ?");
stm.setLong(1, chatId);
ResultSet results = stm.executeQuery();
if(!results.isBeforeFirst()) return null;
while (results.next()){
commands.add(new CustomCommand(results.getString(1),results.getString(2),results.getString(3), CustomCommand.Type.valueOf(results.getString(4)),results.getInt(5)));
}
return commands;
}
return null;
}
Maybe the problem is something related to exception catching and statements not being correctly executed? Maybe something related to result sets?
It's driving me crazy. Thanks for helping me!

You do nothing to clean up ResultSet and Statement before you return. That's a bad idea. You should be closing each one in individual try/catch blocks in a finally block.
A ResultSet is an object that represents a database cursor on the database. You should close it so you don't run out of cursors.
I wouldn't have a single static Connection. I'd expect a thread-safe, managed pool of connections.
I wouldn't return a null. You don't make clear what the user is supposed to do with that. Better to throw an exception.

maximum open cursors exceeded exception in java code

this my code to execute update query
public boolean executeQuery(Connection con,String query) throws SQLException
{
boolean flag=false;
try
{
Statement st = con.createStatement();
flag=st.execute(query);
st.close();
st=null;
flag=true;
}
catch (Exception e)
{
flag=false;
e.printStackTrace();
throw new SQLException(" UNABLE TO FETCH INSERT");
}
return flag;
}
maximum open cursor is set to 4000
code is executing
update tableA set colA ='x',lst_upd_date = trunc(sysdate) where trunc(date) = to_date('"+date+"','dd-mm-yyyy')
update query for around 8000 times
but after around 2000 days its throwing exception as "maximum open cursors exceeded"
please suggest code changes for this.
#TimBiegeleisen here is the code get connecttion
public Connection getConnection(String sessId)
{
Connection connection=null;
setLastAccessed(System.currentTimeMillis());
connection=(Connection)sessionCon.get(sessId);
try
{
if(connection==null || connection.isClosed() )
{
if ( ds == null )
{
InitialContext ic = new InitialContext();
ds = (DataSource) ic.lookup("java:comp/env/iislDB");
}
connection=ds.getConnection();
sessionCon.put(sessId, connection);
}
}
catch (SQLException e)
{
e.printStackTrace();
}
catch (Exception e)
{
e.printStackTrace();
}
return connection;
}
`
error stack is as bellow
java.sql.SQLException: ORA-01000: maximum open cursors exceeded
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:180)
at oracle.jdbc.ttc7.TTIoer.processError(TTIoer.java:208)
at oracle.jdbc.ttc7.Oopen.receive(Oopen.java:118)
at oracle.jdbc.ttc7.TTC7Protocol.open(TTC7Protocol.java:472)
at oracle.jdbc.driver.OracleStatement.<init>(OracleStatement.java:499)
at oracle.jdbc.driver.OracleConnection.privateCreateStatement(OracleConnection.java:683)
at oracle.jdbc.driver.OracleConnection.createStatement(OracleConnection.java:560)
at org.apache.tomcat.dbcp.dbcp.DelegatingConnection.createStatement(DelegatingConnection.java:257)
at org.apache.tomcat.dbcp.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.createStatement(PoolingDataSource.java:216)
at com.iisl.business.adminbo.computeindex.MoviIndexComputeBO.calculateMoviValue(MoviIndexComputeBO.java:230)

Your code has a cursor leak. That's what is causing the error. It seems unlikely that your code can really go 2000 days (about 5.5 years) before encountering the error. If that was the case, I'd wager that you'd be more than happy to restart a server twice a decade.
In your try block, you create a Statement. If an exception is thrown between the time that the statement is created and the time that st.close() is called, your code will leave the statement open and you will have leaked a cursor. Once a session has leaked 4000 cursors, you'll get the error. Increasing max_open_cursors will merely delay when the error occurs, it won't fix the underlying problem.
The underlying problem is that your try/ catch block needs a finally that closes the Statement if it was left open by the try. For this to work, you'd need to declare st outside of the try
finally {
if (st != null) {
st.close();
}
}

As mentioned in another response you will leak cursors if an exception is thrown during the statement execution because st.close() won't be executed. You can use Java's try-with-resources syntax to be sure that your statement object is closed:
try (Statement st = con.createStatement())
{
flag=st.execute(query);
flag=true;
}
catch (Exception e)
{
flag=false;
e.printStackTrace();
throw new SQLException(" UNABLE TO FETCH INSERT");
}
return flag;

One of quickest solution is to increase cursor that each connection can handle by issuing following command on SQL prompt:
alter system set open_cursors = 1000
Also, add finally block in your code and close the connection to help closing cursors when ever exception occurs.
Also, run this query to see where actually cursor are opened.
select sid ,sql_text, count(*) as "OPEN CURSORS", USER_NAME from v$open_cursor
finally {
if (connection!=null) {
connection.close();
}

What is causing the "near ".": syntax error" in my SQLite query?

I am trying to output my table as a .csv file. I am using the sqlite-jdbc java library to execute my query. Here are the statements:
.mode csv
.output test.csv
select * from geninfo;
.output stdout
and this is my java code:
try {
try {
// create a database connection
Class.forName("org.sqlite.JDBC");
} catch (ClassNotFoundException ex) {
Logger.getLogger(SQLite.class.getName()).log(Level.SEVERE, null, ex);
}
connection = DriverManager.getConnection("jdbc:sqlite:702Data.db");
Statement statement = connection.createStatement();
statement.setQueryTimeout(30); // set timeout to 30 sec.
statement.executeUpdate("create table if not exists geninfo (id TEXT, DeviceID TEXT)");
statement.executeUpdate(".mode csv");
statement.executeUpdate(".output test.csv");
statement.executeUpdate("select * from geninfo;");
statement.executeUpdate(".output stdout");
ResultSet rs = statement.executeQuery("select * from geninfo");
while (rs.next()) {
// read the result set
System.out.println("DeviceID = " + rs.getString("DeviceID"));
System.out.println("id = " + rs.getInt("id"));
}
} catch (SQLException e) {
// if the error message is "out of memory",
// it probably means no database file is found
System.err.println(e.getMessage());
} finally {
try {
if (connection != null) {
connection.close();
}
} catch (SQLException e) {
// connection close failed.
System.err.println(e);
}
}
}
I have tried adding parenthesis around the period and also removing the period and both cause the same syntax error.

Those commands are SQLite console commands, not pure SQL. JDBC can only handle SQL.
If you want to do this from java, try executing the shell command to open the console using ProcessBuilder and feed it the commands via its InputStream.
However, if calling from java is not necessary, you may be better served writing a shell script of some sort and using that directly, or calling the script from java.
If all you are using SQLite for is to parse the CSV file, consider not using SQLite at all and instead load the data directly from the file into your code using one of the several open source CSV parsing libraries out there. This approach will be order of magnitude faster and simpler.

Too many open files error

Introduction
I've found plenty of information about the Too many open files exception on the Web but I couldn't solve this strange case. As I've read, the exception is thrown when the number of opened file descriptors by process defined in the OS is exceeded. The nature of these files is diverse. Files could be sockets, documents, etc. And I've found robust and secure ways to open files that I have implemented in my Java application.
The application is a short program that downloads Web pages using the Boilerpipe algorithm. This way I get the most representative contents of that site. Then, I write it in an appropriate format (TREC format) to disk. The URLs of these websites are taken from a MySQL database that I access using the JDBC connector.
So, I think that the exception can be thrown form three different places:
Connection to database
HTTP Connection to the web sites
Opening and writing files
Although, as I said, I think that I use a correct way of opening and writing those files.
Problem
There are thousands of URL's to process and the exception is thrown after a while (what makes it also very difficult to debug...). I don't know if that matters, but URLs are classified into different categories and I run different instances of the program to speed up the whole process. Categories don't overlap so there shouldn't be any problem.
Code
To make it more readable I'm going to show just those three parts of my code simplified:
Database access
// Connect to database
Connection dbconn = null;
try {
String dbUrl = "jdbc:mysql://" + dbServer + "/" + dbName;
Class.forName ("com.mysql.jdbc.Driver").newInstance ();
dbconn = DriverManager.getConnection(dbUrl, dbUser, dbPass);
System.out.println ("Database connection established");
} catch (Exception e) {
e.printStackTrace();
System.err.println ("Cannot connect to database server");
System.exit(-1);
}
System.out.println(" Downloading category: " + category);
Statement s = null;
try {
s = dbconn.createStatement();
} catch (SQLException e) {
System.err.println ("Error on creating the statement");
System.exit(-1);
e.printStackTrace();
}
String q = "SELECT resource,topic FROM " +
"content_links " +
"WHERE topic LIKE 'Top/" + category + "%';";
try {
s.executeQuery(q);
} catch(Exception e) {
System.err.println ("Error on executing the SQL statement");
System.exit(-1);
e.printStackTrace();
}
ResultSet rs = null;
try {
rs = s.getResultSet ();
} catch (SQLException e) {
System.err.println ("Error on getting the result set");
System.exit(-1);
e.printStackTrace();
}
int count = 0, webError = 0;
// work with the result set
try {
while (rs.next ()) {
// MAIN LOOP
}
} catch (SQLException e) {
System.err.println ("Error on getting next item");
System.exit(-1);
e.printStackTrace();
}
// Close connection to database
if (dbconn != null) {
try {
dbconn.close ();
System.out.println (" Database connection terminated");
} catch (Exception e) { /* ignore close errors */ }
}
HTTP connection, extract site's title and boilerpipe filter
try {
String title = "";
org.jsoup.nodes.Document doc = Jsoup.connect(urlVal).get();
for (Element element : doc.select("*")) {
if (element.tagName().equalsIgnoreCase("title")) {
title = element.text();
}
if (!element.hasText() && element.isBlock()) {
element.remove();
}
}
String contents = "";
contents = NumWordsRulesExtractor.INSTANCE.getText(doc.text());
storeFile(id, urlVal, catVal, title, contents);
}
} catch (BoilerpipeProcessingException e) {
System.err.println("Connection failed to: " + urlVal);
} catch (MalformedURLException e1) {
System.err.println("Malformed URL: " + urlVal);
} catch(Exception e2) {
System.err.println("Exception: " + e2.getMessage());
e2.getStackTrace();
}
Writing file
private static void storeFile(String id, String url, String cat, String title, String contents) {
BufferedWriter out = null;
try {
out = new BufferedWriter(
new OutputStreamWriter(
new FileOutputStream(
new File(path + "/" + id + ".webtrec")),"UTF8"));
// write in TREC format
out.write("...");
} catch (IOException e) {
System.err.println("Error: " + e.getMessage());
e.printStackTrace();
} finally {
try {
out.close();
} catch (IOException e) {
System.err.println("Error: " + e.getMessage());
e.printStackTrace();
}
}

Yup. You are leaking file descriptors.
In the first case you open a DB connection and never close it. The connection will typically use a socket or something to talk to the database. Since you don't close the connection, the socket won't be closed, and you will leak a file descriptor.
In the second case, I suspect that the call to Jsoup.connect(urlVal) is opening a connection, which you don't then close. That will result in a file descriptor leak.
Correction - there is no close() method on the Connection interface. It looks like the actual connection must be created and then closed internally by the get method. Assuming that is so, there is no file descriptor leak in the second case.
The third case does not leak file descriptors. However, if you fail to open the file, out.close(); statement will attempt to call a method on null ... and will throw a NPE.
The solution is to find all of the places where you open files, database connection, http connections, and make sure that the handle is always closed.
One way to do it is to put the close() call (or equivalent) in a finally block ... but make sure that you don't accidentally call close() on null.
The other way to do it is to use the Java 7 "try with resource" syntax. For example:
private static void storeFile(String id, String url, String cat,
String title, String contents) {
try (BufferedWriter out = new BufferedWriter(
new OutputStreamWriter(
new FileOutputStream(
new File(path + "/" + id + ".webtrec")),"UTF8"))) {
// write in TREC format
out.write("...");
out.close();
} catch (IOException e) {
System.err.println("Error: " + e.getMessage());
e.printStackTrace();
}
}
(Note however that the Java 7 syntax can only be used with resources that implement the new Closeable interface.)

To add to Stephen's comprehensive analysis. I recommend using a connection pool for the database, although, as Stephen has pointed, unless you're closing these connections, you'll drain the pool close, but at least it will be easier to discover why...
I've not seen any evidence, but you should be using some kind of Thread pool to download the pages, this will help to maximize the resources of the system. Some of executor service would suffice. Like I say, you're probably already doing this, but you didn't show any code (or comment) for it.

Will Database omit incoming request since it is busy?

In my application I have implemented a method to get favourits of particular user. If the user is a new one there will not be a entry in the table.If so I add default favourtis to the table. Code is shown below.
public String getUserFavourits(String username) {
String s = "SELECT FAVOURITS FROM USERFAVOURITS WHERE USERID='" +
username.trim() + "'";
String a = "";
Statement stm = null;
ResultSet reset = null;
DatabaseConnectionHandler handler = null;
Connection conn = null;
try {
handler = DatabaseConnectionHandler.getInstance();
conn = handler.getConnection();
stm = conn.createStatement(ResultSet.TYPE_SCROLL_SENSITIVE,ResultSet.CONCUR_UPDATABLE);
reset = stm.executeQuery(s);
if (reset.next()) {
a = reset.getString("FAVOURITS").toString();
}
reset.close();
stm.close();
}
catch (SQLException ex) {
ex.printStackTrace();
}
catch (Exception ex) {
ex.printStackTrace();
}
finally {
try {
handler.returnConnectionToPool(conn);
if (stm != null) {
stm.close();
}
if (reset != null) {
reset.close();
}
}catch (Exception ex) {
ex.printStackTrace();
}
}
if (a.equalsIgnoreCase("")) {
a = updateNewUserFav(username);
}
return a;
}
You can see that after the Finally block updateNewUserFav(username) method is use to insert default favourits in to table. Normally users are forced to change this in their first login.
My problem is many users have complain me about they hava lost their customized favourits and default has get loaded in their login. When I go through the code I notice that it can only happen if exception occured in the try block. When I debug code works fine. Is this can be coused at time when DB is busy?
Normally there are more than 1000 concurrent user in the system. Since it is real time application there will be huge number a of request comming to the Database(DB is Oracle).
Can some one pls explain.

Firstly, use jonearles suggestion about bind variables. If a lot of your code is like this, with 1000 concurrent users, I'd hate to think what performance is like.
Secondly, if it is busy then there is a chance of time-outs. As you say, if an exception is encountered then it falls back to the "updateNewUserFav"
Really, it should only call that if NO exception is raised.
If an exception is raised, the function should fail. The current code is similar to
"TURN THE IGNITION KEY TO START THE CAR"
"IF THERE IS A PROBLEM, RING GARAGE AND BOOK APPOINTMENT"
"PUT CAR INTO GEAR AND RELEASE HAND_BRAKE"
You really only want to release the hand-brake once the car has successfully started, otherwise you'll end up rolling down the hill until the sudden stop at the end (often involving an expensive CRUNCH sound).