Develop Reference

Java SQL memory leak

I'm facing an issue where I have a java application running on a server, and it starts growing in memory until eventually the server cannot handle it anymore.
This is some sort of memory leak / resource leak problem, which I thought was extremely rare in Java due to the garbage collection. I guess something is being referenced and never used, so the garbage collector does not collect it.
The problem is that the size in memory grows so slowly that I'm not able to debug it properly (it may take two weeks to make the server unusable).
I'm using java + mysql-connector, and I'm sure the memory leak is caused by something related to the database connection.
Here is how I connect to the database:
private static Connection connect(){
try {
Connection conn = null;
conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/database","client","password");
return conn;
}catch(SQLException ex){
System.out.println("SQLException: " + ex.getMessage());
System.out.println("SQLState: " + ex.getSQLState());
System.out.println("VendorError: " + ex.getErrorCode());
return null;
}
}
public static Connection getConnection(){
try {
if (connection == null || connection.isClosed()) connection = connect();
return connection;
}catch (SQLException exception){
System.out.println("exception trying to connect to the database");
return null;
}
}
I can't find any possible problem here, but who knows!
Here's how I retrieve information from the database:
public void addPoints(long userId,int cantidad){
try {
if(DatabaseConnector.getConnection()!=null) {
PreparedStatement stm = DatabaseConnector.getConnection().prepareStatement("UPDATE users SET points = points + ? WHERE id = ? ");
stm.setLong(2, userId);
stm.setInt(1, cantidad);
if(stm.executeUpdate()==0){ //user doesn't have any point records in the database yet
PreparedStatement stm2 = DatabaseConnector.getConnection().prepareStatement("INSERT INTO users (id,points) VALUES (?,?)");
stm2.setLong(1, userId);
stm2.setInt(2, cantidad);
stm2.executeUpdate();
}
}
}catch (SQLException exception){
System.out.println("error recording points");
}
}
public ArrayList<CustomCommand> getCommands(long chatId) throws SQLException{
ArrayList<CustomCommand> commands = new ArrayList<>();
if(DatabaseConnector.getConnection() != null) {
PreparedStatement stm = DatabaseConnector.getConnection().prepareStatement("SELECT text,fileID,commandText,type,probability FROM customcommands WHERE chatid = ?");
stm.setLong(1, chatId);
ResultSet results = stm.executeQuery();
if(!results.isBeforeFirst()) return null;
while (results.next()){
commands.add(new CustomCommand(results.getString(1),results.getString(2),results.getString(3), CustomCommand.Type.valueOf(results.getString(4)),results.getInt(5)));
}
return commands;
}
return null;
}
Maybe the problem is something related to exception catching and statements not being correctly executed? Maybe something related to result sets?
It's driving me crazy. Thanks for helping me!

You do nothing to clean up ResultSet and Statement before you return. That's a bad idea. You should be closing each one in individual try/catch blocks in a finally block.
A ResultSet is an object that represents a database cursor on the database. You should close it so you don't run out of cursors.
I wouldn't have a single static Connection. I'd expect a thread-safe, managed pool of connections.
I wouldn't return a null. You don't make clear what the user is supposed to do with that. Better to throw an exception.

Closes connection and creates a new one not to exceed max cursors after 295 inserts

A previous developer left this message in code.
It runs create statement and execute query and after 295 records it creates new connection.
Following is java code:
private void dbUpdate() throws SQLException, Exception {
Statement st = null;
String sql = "";
int count = 0;
try {
getNewConnection();
conn.setAutoCommit(false);
for (Iterator it = sqlList.iterator(); it.hasNext();) {
if (count < 295) { //Closes connection and creates a new one so as not to exceed max cursors
count++;
} else {
st.close();
conn.close();
getNewConnection();
count = 0;
}
sql = (String) it.next();
// System.out.println(sql + " insert count=" + count);
st = conn.createStatement();
try {
st.executeQuery(sql);
} catch(Exception ex) {
Logger.getLogger(LoadMain.class.getName()).log(Level.SEVERE, sql);
Logger.getLogger(LoadMain.class.getName()).log(Level.SEVERE, ex.getLocalizedMessage(), ex);
sb.append("\n").append("Error SQL:" + sql + "|LocalizedMessage:" +ex.getLocalizedMessage());
}
}
} catch (SQLException ex) {
Logger.getLogger(LoadMain.class.getName()).log(Level.INFO, sql);
Logger.getLogger(LoadMain.class.getName()).log(Level.SEVERE, ex.getLocalizedMessage(), ex);
throw new SQLException(ex);
} catch (Exception ex) {
// Logger.getLogger(loadMain.class.getName()).log(Level.SEVERE, ex.getLocalizedMessage(), ex);
throw new Exception(ex);
} finally {
try {
st.close();
conn.close();
} catch (SQLException ex) {
Logger.getLogger(LoadMain.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
Is there any logic behind re connection?
(Also developer set autocommit false but not seen committing or roll-backing but only st.close() methods.)
Could anyone please enlighten

Seems as the developer tried to implement connection pooling, which now can be integrated easily with DBCP/Hikari/other database connection pool.
You don't need to commit on DAO level/method, for example
if the code is called by a method with #Transactional or commit is handled on service level.
Also you can't rely that close will commit or rollback, there can be different results with different oracle drivers
According to the javadoc, you should try to either commit or roll back before calling the close method. The results otherwise are implementation-defined.

MySql database connection issues in play framework (driver not found)

So I have tried using the stock Play! 2.2 configuration for the MySql database connection. Unfortunately the guides out there are less than helpful when using the stock database (h2) alongside a MySql. SO, I coded a separate model to handle the MySql connection. It works intermittently, and I'm trying to figure out why it doesn't work all of the time.
this is the "connect" function
String sourceSchema = "db";
String databaseHost = "host";
String databaseURLSource = "jdbc:mysql://" + databaseHost + "/" + sourceSchema;
String databaseUserIDSource = "userid";
String databasePWDSource = "password";
try {
Class.forName("com.mysql.jdbc.Driver").newInstance();
conn = DriverManager.getConnection(databaseURLSource,
databaseUserIDSource, databasePWDSource);
return true;
} catch (InstantiationException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IllegalAccessException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (SQLException e) {
Logger.error("SQLException: " + e.getMessage());
}
All of my credentials are correct (here obviously they are changed) Next, in my lib folder, I have the
mysql-connector-java-5.1.21-bin.jar
in place.
Next, in my Build.scala, I have this under appDependencies:
"mysql" % "mysql-connector-java" % "5.1.21"
when I try to validate the connection, using:
public boolean isConnected() {
return conn != null;
}
The connection fails (intermittantly) and then gives me:
SQLException: Before start of result set
and sometimes:
SQLException: No Suitable driver found for mysql ...
This is how my query is executed:
String qs = String.format("SELECT * FROM community_hub.alert_journal LIMIT("+ from +","+ to +")");
String qscount = String.format("SELECT COUNT(*) AS count FROM community_hub.alert_journal");
try {
if (isConnected()) {
Statement stmt = conn.createStatement();
//obtain count of rows
ResultSet rs1 = stmt.executeQuery(qscount);
//returns the number of pages to draw on index
int numPages = returnPages(rs1.getInt("count"),rpp);
NumPages(numPages);
ResultSet rs = stmt.executeQuery(qs);
while (rs.next())
{
AlertEntry ae = new AlertEntry(
rs.getTimestamp("date"),
rs.getString("service_url"),
rs.getString("type"),
rs.getString("offering_id"),
rs.getString("observed_property"),
rs.getString("detail")
);
list.add(ae);
}
rs.close();
disconnect();
} else {
System.err.println("Connection was null");
}
}
catch (Exception e)
{
e.printStackTrace();
}
Help?
Thanks!

does the mysql error tell you anything?
the first error "SQLException: Before start of result set" looks like its incomplete. Maybe the error log contains the full message or you can
the second one "SQLException: No Suitable driver found for mysql" clearly indicates a classpath issue.
usually connection pools like c3p0 or BoneCP recommed to use a validation query to determine if a connection is valid (something like "select 1" for mysql). That may help to make sure the connection is ok and not rely on the driver?

Java SQL Optimization

I am trying to use an SQL database with a Java program. I make a table that is 7 columns wide and 2.5 million rows (My next one I need to build will be about 200 million rows). I have two problems: building the SQL table is too slow (about 2,000 rows/minute) and searching the database is too slow (I need to find over 100 million rows in under a second if possible, it currently takes over a minute). I have tried creating a csv file and importing it, but I can't get it to work.
I am using xampp and phpMyAdmin on my computer (i5 + 6gb ram). I have three methods I am testing: createTable(), writeSQL(), and searchSQL().
createTable:
public static void createTable() {
String driverName = "org.gjt.mm.mysql.Driver";
Connection connection = null;
try {
Class.forName(driverName);
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
String serverName = "localhost";
String mydatabase = "PokerRanks4";
String url = "jdbc:mysql://" + serverName + "/" + mydatabase;
String username = "root";
String password = "";
try {
connection = DriverManager.getConnection(url, username, password);
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
///////////////
String table = "CREATE TABLE ranks(deckForm bigint(10) NOT NULL,rank0 int(2) NOT NULL,rank1 int(2) NOT NULL,rank2 int(2) NOT NULL,rank3 int(2) NOT NULL,rank4 int(2) NOT NULL,rank5 int(2) NOT NULL,PRIMARY KEY (deckForm),UNIQUE id (deckForm),KEY id_2 (deckForm))";
try {
Statement st = connection.createStatement();
st.executeUpdate(table);
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
///////////////
try {
connection.close();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
writeSQL():
public static void writeSQL() {
String driverName = "org.gjt.mm.mysql.Driver";
Connection connection = null;
try {
Class.forName(driverName);
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
String serverName = "localhost";
String mydatabase = "PokerRanks4";
String url = "jdbc:mysql://" + serverName + "/" + mydatabase;
String username = "root";
String password = "";
try {
connection = DriverManager.getConnection(url, username, password);
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
/////////////// Prepared Statement with Batch
PreparedStatement statement = null;
String sql = "INSERT INTO ranks VALUES (? ,0, 0, 0, 0, 0, 0)";
long start = System.currentTimeMillis();
try {
statement = connection.prepareStatement(sql);
for (int i = 0; i < 100; i++) {
for (int j = 0; j < 100; j++) {
statement.setLong(1, (i*100 + j));
statement.addBatch();
}
System.out.println(i);
statement.executeBatch();
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (statement != null) {
try {
statement.close();
} catch (SQLException e) {
} // nothing we can do
}
if (connection != null) {
try {
connection.close();
} catch (SQLException e) {
} // nothing we can do
}
}
System.out.println("Total Time: " + (System.currentTimeMillis() - start) / 1000 );
///////////////
}
searchSQL():
public static void searchSQL() {
String driverName = "org.gjt.mm.mysql.Driver";
Connection connection = null;
try {
Class.forName(driverName);
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
String serverName = "localhost";
String mydatabase = "PokerRanks2";
String url = "jdbc:mysql://" + serverName + "/" + mydatabase;
String username = "root";
String password = "";
try {
connection = DriverManager.getConnection(url, username, password);
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
/////////////// Option 1, Prepared Statement
ResultSet rs = null;
PreparedStatement pstmt = null;
String query = "SELECT rank0, rank1, rank2, rank3, rank4, rank5 FROM ranks WHERE deckForm = ?";
long start = System.currentTimeMillis();
try {
pstmt = connection.prepareStatement(query);
for (int i = 0; i < 100000; i++) {
pstmt.setLong(1, 1423354957);
rs = pstmt.executeQuery();
while (rs.next()) {
int[] arr = {rs.getInt(1), rs.getInt(2), rs.getInt(3), rs.getInt(4), rs.getInt(5), rs.getInt(6)};
}
}
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println("Total Time: " + (System.currentTimeMillis() - start) / 1000 );
///////////////
/*
/////////////// Option 2
Statement st = null;
long start = System.currentTimeMillis();
try {
st = connection.createStatement();
ResultSet rs = null;
long deckForm = 1012213456;
for (int i = 0; i < 100000; i++) {
rs = st.executeQuery("SELECT rank0, rank1, rank2, rank3, rank4, rank5 FROM ranks WHERE deckForm = " + deckForm);
while (rs.next()) {
int[] arr = {rs.getInt(1), rs.getInt(2), rs.getInt(3), rs.getInt(4), rs.getInt(5), rs.getInt(6)};
}
}
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println("Total Time: " + (System.currentTimeMillis() - start) / 1000 );
///////////////
*/
try {
connection.close();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Sorry that's so long. I've tried everything I can think of to make this faster but I can't figure it out. Any suggestions?

Well, there's a few improvements you could make:
You are creating a connection each time you want to search, write or create,
you should use a pooled connection and datasources.
Optimize your queries by doing explain plans, and optimize your table relations and indexes.
You can use stored procedures and call them.
Well that's all I can help with, certainly there are more tips.

As to the insert speed, you need to disable all the indexes prior to doing the insert and re-enable them after you're done. Please see Speed of Insert Statements for a lot of detailed information on improving bulk insert speed.
The query speed is probably limited by your CPU and disk speeds. You may have to throw much more hardware at the problem.

building the SQL table is too slow (about 2,000 rows/minute)
So point of view on inserting a great number of rows is sure use Heap table, it's basic table, also it named as persistent page-array usually created just by CREATE TABLE, it's not effective for searching as you meant that search is slow but for inserting is very efficient because it add rows to first free position that what find or on the end of table. But on other hand, searching is very inefficietly, because is not guaranteed sort of items/rows.
searching the database is too slow (I need to find over 100 million
rows in under a second if possible, it currently takes over a minute)
So for this you should create table in that is searching is efficiently. In a case if you using Oracle, so it offers many constructions for physical implementation for example Index organized tables, Data clustering, Clustered tables - Index / Hash / Sorted hash ...
SQL Server i'm not sure but also clustered tables and MySQL i don't know exactly, i don't want to tell you something worst. I don't say that MySQL is bad or worse like Oracle for example but just not offer some techniques for physical implementation like Oracle for example
So, i mean that it's quite hard to say some recommendations for this approach but you seriously think and study something about physical implementations of database systems, have look at relational algebra for optimize your statements, which types of tables you should create, #duffymo meant right that you can let explain your query execution plan by EXPLAIN PLANE FOR and based on result to optimize. Also how to use indexes, it's strong database construction but each index mean much more operations for any modifying of database so well to rethink for which attribute you create index etc.
Via Google, you find many useful articles about data modeling, physical implementation etc.
Regards man, I wish best of luck

Hitting stored proc - javax.ejb.EJBException

I'm trying to hit a stored procedure but I'm getting this error message: 'javax.ejb.EJBException'... I've never worked with stored procedures so the exception is a bit Greek to me.
Anyone that could perhaps shed some light on this? Below I pasted the code that I wrote:
#WebMethod(operationName = "getSpecimenResultsXml")
public String getSpecimenResultsXml(#WebParam(name = "specimenGuid") String specimenGuid, #WebParam(name = "publicationGuid") String publicationGuid, #WebParam(name = "forProvider") String forProvider) {
//Method variables
ResultSet rs = null;
String xml = null;
// 1) get server connection
Connection conn = dataBaseConnection.getConnection();
// 2) Pass recieved parameters to stored proc.
try {
CallableStatement proc =
conn.prepareCall("{ call getSpecimenReportXml(?, ?, ?) }");
proc.setString(1, specimenGuid);
proc.setString(2, publicationGuid);
proc.setString(3, forProvider);
proc.execute();
rs = proc.getResultSet();
} catch (SQLException e) {
System.out.println("--------------Error in getSpecimenResultsXml------------");
System.out.println("Cannot call stored proc: " + e);
System.out.println("--------------------------------------------------------");
}
// 3) Get String from result set
try {
xml = rs.getString(1);
} catch (SQLException e) {
System.out.println("--------------Error in getSpecimenResultsXml------------");
System.out.println("Cannot retrieve result set: " + e);
System.out.println("--------------------------------------------------------");
}
// 4) close connection
try {
conn.close();
} catch (Exception e) {
System.out.println("--------------Error in getSpecimenResultsXml------------");
System.out.println("Cannot close connection: " + e);
System.out.println("--------------------------------------------------------");
}
// 5) return the returned String
return xml;
}
Oh and the stored procedure us called getSpecimenReportXml...

Your exception would say 'caused by' somewhere - which is a big clue. If it's an NPE then you might want to check the values of dataBaseConnection and conn to make sure they've been set. Use a debugger to do this, but the exception should tell you exactly which line caused the problem.