Develop Reference

Unexpected memory using when using Postgres jdbc driver

I am trying to execute a query using postgre sql driver for java jdbc.
I have an issue with memory buildup my statement is in a loop and then sleeps.
The problem is when I look at the job in task manager I can see the memory climbing 00,004K at a time. I have read the documentation I have closed all connections statements resultsets but this still happens.
Please could you tell me what is causing this in my code.
String sSqlString = new String("SELECT * FROM stk.comms_data_sent_recv " +
"WHERE msgtype ='RECEIVE' AND msgstat ='UNPRC' " +
"ORDER BY p6_id,msgoccid " +
"ASC; ");
ResultSet rs = null;
Class.forName("org.postgresql.Driver");
Connection connection = DriverManager.getConnection(
"jdbc:postgresql://p6tstc01:5432/DEVC_StockList?autoCloseUnclosedStatements=true&logUnclosedConnections=true&preparedStatementCacheQueries=0&ApplicationName=P6Shunter", "P6dev",
"admin123");
//Main Loop
while(true)
{
try{
Statement statement = connection.createStatement();
statement.executeQuery(sSqlString);
//rs.close();
statement.close();
//connection.close();
rs = null;
//connection = null;
statement =null;
}
finally {
//connection.close();
}
try {
Thread.sleep(loopTime);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Notice the commented out code.. I did close all but that did not seem to make a difference. Whet I did see is that it seems that the statement executeQuery(sSqlString); is causing this the reason I think so is if I remove the statement there is no memory leak.
I could be wrong but please assist me.
UPDATE:
I have changed my code as with your recommendations. Hope its a bit better please let me know if I need to change something.
My main loop :
public static void main(String[] args) throws Exception {
// TODO Auto-generated method stub
//Main Loop
while(true)
{
getAndProcessAllUnprcMessagesFromStockList();
try {
Thread.sleep(loopTime);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
My Function it will call do fetch data :
public static void getAndProcessAllUnprcMessagesFromStockList() throws Exception
{
ResultSet rs = null;
Statement statement = null;
Connection connection =null;
String sSqlString = new String("SELECT * FROM stk.comms_data_sent_recv " +
"WHERE msgtype ='RECEIVE' AND msgstat ='UNPRC' " +
"ORDER BY p6_id,msgoccid " +
"ASC; ");
try{
Class.forName("org.postgresql.Driver");
connection = DriverManager.getConnection(
"jdbc:postgresql://p6tstc01:5432/DEVC_StockList?autoCloseUnclosedStatements=true&logUnclosedConnections=true&preparedStatementCacheQueries=0&ApplicationName=P6Shunter", "P6dev",
"admin123");
PreparedStatement s = connection.prepareStatement(sSqlString,
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
rs = s.executeQuery();
while (rs.next()) {
//Process records
UnprcMsg msg = new UnprcMsg();
msg.setP6Id(rs.getString(1));
msg.setMsgOccId(rs.getString(2));
msg.setWsc(rs.getString(3));
msg.setMsgId(rs.getString(4));
msg.setMsgType(rs.getString(5));
msg.setMsgStatus(rs.getString(6));
//JOptionPane.showMessageDialog(null,msg.getP6Id(), "InfoBox: " + "StockListShunter", JOptionPane.INFORMATION_MESSAGE);
//msg2 = null;
}
rs.close();
s.close();
}
catch(Exception e)
{
e.printStackTrace();
}
finally
{
connection.close();
}
}
I have closed my connections statements and results.
I also downloaded eclipse memory analyzer and I ran the jar witch will execute my main loop. Ran it for about an hour and here's some of the data I got from memory analyzer..
Leak suspects :
Now I know I cant go on the memory usage of task manager but whats the difference? Why does task manager show the following :
I was concerned about the memory usage I see in task manager? should I be?

Database rows reading into list of objects

public static List<SPACE_CreateLicenseModel> SPACE_getDetails() throws ClassNotFoundException, FileNotFoundException, JSONException{
SPACE_CreateLicenseModel view = new SPACE_CreateLicenseModel();
Statement stmt = null;
Connection connect = null;
List<SPACE_CreateLicenseModel> allData = new ArrayList<SPACE_CreateLicenseModel>();
try {
connect = SPACE_DBController.SPACE_getConnection();
stmt = connect.createStatement();
JSONObject obj = SPACE_Parse.parse ("C:/Users/Rachana/workspace/SPACEOM/WebContent/Data/SPACE_Database.json");
String tablename = obj.getString("table_name");
String sql = "SELECT * FROM " + tablename + " WHERE (SPLD_LicenseActiveStatus <> 5 OR SPLD_LicenseActiveStatus IS NULL)";
ResultSet result = stmt.executeQuery(sql);
int i =0;
while (result.next()) {
view.setSPLD_DeviceID_Mfg(result.getString(1));
view.setSPLD_DeviceID_ModelNo(result.getString(2));
view.setSPLD_DeviceID_SrNo(result.getString(3));
view.setSPLD_DeviceID_Search_mode(result.getByte(4));
view.setSPLD_LicenseType(result.getByte(5));
view.setSPLD_LicenseTypeChangedDate(result.getDate(6));
view.setSPLD_LicenseActiveStatus(result.getByte(7));
view.setSPLD_LicenseActiveDate(result.getDate(8));
view.setSPLD_LicenseAccess(result.getByte(9));
view.setSPLD_LicenseAccessMaxNo(result.getInt(10));
view.setSPLD_LicenseAccessCounter(result.getInt(11));
view.setSPLD_LicenseStartDate(result.getDate(12));
view.setSPLD_LicenseExpiryDate(result.getDate(13));
view.setSPLD_LicenseeOrg(result.getString(14));
view.setSPLD_LicenseeAddress(result.getString(15));
view.setSPLD_LocationActive(result.getString(16));
view.setSPDL_Longitude(result.getDouble(17));
view.setSPDL_Latitude(result.getDouble(18));
view.setSPDL_LocationTolerance(result.getFloat(19));
view.setSPLD_FutureOption1(result.getString(20));
view.setSPLD_FutureOption2(result.getString(21));
view.setSPLD_FutureOption3(result.getString(22));
view.setSPLD_FutureOption4(result.getInt(23));
view.setSPLD_FutureOption5(result.getInt(24));
view.setSPLD_StatCounter1_FirstUseDate(result.getDate(25));
view.setSPLD_StatCounter2_MessageTotal(result.getInt(26));
view.setSPLD_StatCounter3_FailedAttempts(result.getInt(27));
view.setSPLD_StatCounter4_FirstFailedAttemptDate(result.getDate(28));
view.setSPLD_StatCounter5_LastFailedAttemptDate(result.getDate(29));
view.setSPLD_StatCounter6(result.getInt(30));
view.setSPLD_StatCounter7(result.getInt(31));
view.setSPLD_StatCounterOption1(result.getString(32));
view.setSPLD_StatCounterOption2(result.getString(33));
view.setSPLD_StatCounterOption3(result.getString(34));
view.setSPLD_StatCounterOption4(result.getInt(35));
view.setSPLD_StatCounterOption5(result.getInt(36));
view.setSPLD_MainContact1Name(result.getString(37));
view.setSPLD_MainContact2Name(result.getString(38));
view.setSPLD_MobileNo1(result.getString(39));
view.setSPLD_MobileNo2(result.getString(40));
view.setSPLD_EmailID1(result.getString(41));
view.setSPLD_EmailID2(result.getString(42));
view.setSPLD_CustomerDetailOption1(result.getString(43));
view.setSPLD_CustomerDetailOption2(result.getString(44));
view.setSPLD_BroadCastGEN1(result.getString(45));
view.setSPLD_BroadCastGEN2(result.getString(46));
view.setSPLD_BroadCastID1(result.getInt(47));
view.setSPLD_DevSpecGEN1(result.getString(48));
view.setSPLD_DevSpecGEN2(result.getString(49));
view.setSPLD_DevSpecGEN3(result.getString(50));
view.setSPLD_DevSpecID1(result.getInt(51));
view.setSPLD_DevSpecID2(result.getInt(52));
view.setSPLD_MessageStatus(result.getString(53).charAt(0));
allData.add(i,view);
i++;
}
} catch (SQLException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
} catch (ClassNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}finally{
//finally block used to close resources
try{
if(stmt!=null)
stmt.close();
}catch(SQLException se2){
}// nothing we can do
try{
if(connect!=null)
connect.close();
}catch(SQLException se){
se.printStackTrace();
}
}
return allData;
}
I am fetching all the rows of the database and storing it in array. But while displaying only the last row is getting printed. The list elements are getting overridden. i.e., allData.add(1,view), allData.add(2,view) , allData.add(3,view) , allData.add(4,view) etc everything are same.

As you are not creating a new Object for each iteration of the loop, it is re-using the same object, so try
Statement stmt = null;
Connection connect = null;
List<SPACE_CreateLicenseModel> allData = new ArrayList<SPACE_CreateLicenseModel>();
try {
connect = SPACE_DBController.SPACE_getConnection();
....
while (result.next()) {
SPACE_CreateLicenseModel view = new SPACE_CreateLicenseModel();

Cause:
Currently for each row same object is getting updated hence all your objects in list have same values (Last Row).
Resolution:
You need to initialize SPACE_CreateLicenseModel each time in loop for every row.
while (result.next()) {
SPACE_CreateLicenseModel view = new SPACE_CreateLicenseModel();
view.setSPLD_DeviceID_Mfg(result.getString(1));
.
.
allData.add(i,view);
i++;
}
Hope this helps

Create a new view object with every iteration of your while loop. Every time you loop through the same view object is getting over written in memory. The final time your loop runs it replaces it with the last row values which is being displayed when you are printing your data...
while(yourCondition){
view = new SPACE_CreateLicenseModel();
//your code goes here....
}
Adding the above line in your loop will create a new view Object and will be added to your allData variable.

How do I perform a simple email/password verification in JSP using a SQL database?

I have the code that successfully establishes a connection to a mySQL database.
String email, password; //assume these are already loaded with user-entered data.
try {
Class.forName("com.mysql.jdbc.Driver");
} catch (ClassNotFoundException e) {
return false;
}
Connection conn = null;
try {
conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/main", "root", "password123");
} catch (SQLException e) {
return false;
}
//perform my database actions here///////////////
///////////////////////////////////////////////////
try {
conn.close();
} catch (SQLException e) {
return false;
}
I have a couple of strings in the scope of the code above that already have the email and password entered by a user on a login page. I need to look through the database for a matching email address and then verify that the password matches what the user entered in the form.
My table has 3 columns: id, email, and password.
I have pushed two rows into the table using the sql workbench
1 | email#gmail.com | password1
2 | email2#gmail.com | password2
I'm assuming in pure SQL I have to do something like
SELECT * FROM users WHERE email LIKE 'email#gmail.com' AND password LIKE 'password1';
But I'm not quite sure how to actually send these SQL commands to the database and receive info back using JSP. Also, I'm not entirely sure my SQL logic is the ideal way to verify a password. My thinking with the SQL command above was that if the database finds any row that meets the conditions, then the email/password combination are verified. Not sure if this is a great way to do it though. I'm not looking for the most secure and complicated way, I'm just looking for the simplest way that makes sense at the moment. Every tutorial I find seems to do it differently and I'm a bit confused.

Here's an example you can use from something I've worked on (I'm assuming that the connection "conn" is obvious):
PreparedStatement st = null;
ResultSet rec = null;
SprayJobItem item = null;
try {
st = conn.prepareStatement("select * from sprayjob where headerref=? and jobname=?");
st.setString(1, request.getParameter("joblistref"));
st.setString(2, request.getParameter("jobname"));
rec = st.executeQuery();
if (rec.next()) {
item = new SprayJobItem(rec);
}
} catch (SQLException ex) {
// handle any errors
ReportError.errorReport("SQLException: " + ex.getMessage());
ReportError.errorReport("SQLState: " + ex.getSQLState());
ReportError.errorReport("VendorError: " + ex.getErrorCode());
} catch (Exception ex) {
ReportError.errorReport("Error: " + ex.getMessage());
} finally {
// Always make sure result sets and statements are closed,
if (ps != null) {
try {
ps.close();
} catch (SQLException e) {
;
}
ps = null;
}
if (rs != null) {
try {
rs.close();
} catch (SQLException e) {
;
}
rs = null;
}
}
In your case instead of item = new SprayJobItem(rec);
you would have code that notes that the user is valid as the record has been found.

Optimizing database inserts java

I am relatively new to java and database and therefore asking your help for my code optimization. I have around 20 text files with comma separated values.Each text files has around 10000 lines Based on the the 3rd value in each line, I insert the data into different tables. Each time I check the 3rd value and use different methods to save this data. My code is as follows. Could someone please tell me if this is the proper way to do this operation.
Thanks in advance.
public void readSave() throws SQLException
{
File dir = new File("C:\\Users\\log");
String url = Config.DB_URL;
String user= Config.DB_USERNAME;
String password= Config.DB_PASSWORD;
con= DriverManager.getConnection(url, user, password);
con.setAutoCommit(false);
String currentLine;
if (!dir.isDirectory())
throw new IllegalStateException();
for (File file : dir.listFiles()) {
BufferedReader br;
try {
br = new BufferedReader(new FileReader(file));
while ((currentLine = br.readLine()) != null) {
List<String> values = Arrays.asList(currentLine.split(","));
if (values.get(2).contentEquals("0051"))
save0051(values,con);
else if(values.get(2).contentEquals("0049"))
save0049(values,con);
else if(values.get(2).contentEquals("0021"))
save0021(values,con);
else if(values.get(2).contentEquals("0089"))
save0089(values,con);
if(statement!=null)
statement.executeBatch();
}
} catch (FileNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
try {
con.commit();
statement.close();
con.close();
}
catch (Exception e) {}
}
private void save0051(List<String> values, Connection connection) throws SQLException {
// TODO Auto-generated method stub
String WRITE_DATA = "INSERT INTO LOCATION_DATA"
+ "(loc_id, timestamp, message_id" +
) VALUES (?,?,?)";
try {
statement = connection.prepareStatement(WRITE_DATA);
statement.setString(1, values.get(0));
statement.setLong(2, Long.valueOf(values.get(1)));
statement.setInt(3, Integer.valueOf(values.get(2)));
statement.addBatch();
} catch (SQLException e) {
e.printStackTrace();
System.out.println("Could not save to DB, error: " + e.getMessage());
}
return;
}

Don't create the database connection in the loop. This is an expensive operation and you should create it only once.
Don't create the PreparedStatement in the loop. Create it once and reuse it.
Don't commit after every single INSERT. Read about using batches for inserting. This reduces the "commit-overhead" dramatically if you only make a commit every let's say 200 INSERTs.

If this is going to be performance critical I'd suggest a few changes.
Move the connection creation out of the loop, you don't want to be doing that thousands of times.
Since each function is repeatedly making the same query, you can cache the PreparedStatements, and repeatedly execute them rather than recreating them with each query. This way the database will only need to optimize the query once, and each query will only transmit the data for the query as opposed to the entire query and the data.

Just spotted that batch insert was already mentioned but here is a nice tutorial page I came across, I think he explains it quite well

Use JDBC Batch INSERT,executeBatch() is faster as insert is made in one shot as a list.
see
http://javarevisited.blogspot.com/2013/01/jdbc-batch-insert-and-update-example-java-prepared-statement.html
Efficient way to do batch INSERTS with JDBC
http://www.java2s.com/Code/Java/Database-SQL-JDBC/BatchUpdateInsert.htm

MySql database connection issues in play framework (driver not found)

So I have tried using the stock Play! 2.2 configuration for the MySql database connection. Unfortunately the guides out there are less than helpful when using the stock database (h2) alongside a MySql. SO, I coded a separate model to handle the MySql connection. It works intermittently, and I'm trying to figure out why it doesn't work all of the time.
this is the "connect" function
String sourceSchema = "db";
String databaseHost = "host";
String databaseURLSource = "jdbc:mysql://" + databaseHost + "/" + sourceSchema;
String databaseUserIDSource = "userid";
String databasePWDSource = "password";
try {
Class.forName("com.mysql.jdbc.Driver").newInstance();
conn = DriverManager.getConnection(databaseURLSource,
databaseUserIDSource, databasePWDSource);
return true;
} catch (InstantiationException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IllegalAccessException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (SQLException e) {
Logger.error("SQLException: " + e.getMessage());
}
All of my credentials are correct (here obviously they are changed) Next, in my lib folder, I have the
mysql-connector-java-5.1.21-bin.jar
in place.
Next, in my Build.scala, I have this under appDependencies:
"mysql" % "mysql-connector-java" % "5.1.21"
when I try to validate the connection, using:
public boolean isConnected() {
return conn != null;
}
The connection fails (intermittantly) and then gives me:
SQLException: Before start of result set
and sometimes:
SQLException: No Suitable driver found for mysql ...
This is how my query is executed:
String qs = String.format("SELECT * FROM community_hub.alert_journal LIMIT("+ from +","+ to +")");
String qscount = String.format("SELECT COUNT(*) AS count FROM community_hub.alert_journal");
try {
if (isConnected()) {
Statement stmt = conn.createStatement();
//obtain count of rows
ResultSet rs1 = stmt.executeQuery(qscount);
//returns the number of pages to draw on index
int numPages = returnPages(rs1.getInt("count"),rpp);
NumPages(numPages);
ResultSet rs = stmt.executeQuery(qs);
while (rs.next())
{
AlertEntry ae = new AlertEntry(
rs.getTimestamp("date"),
rs.getString("service_url"),
rs.getString("type"),
rs.getString("offering_id"),
rs.getString("observed_property"),
rs.getString("detail")
);
list.add(ae);
}
rs.close();
disconnect();
} else {
System.err.println("Connection was null");
}
}
catch (Exception e)
{
e.printStackTrace();
}
Help?
Thanks!

does the mysql error tell you anything?
the first error "SQLException: Before start of result set" looks like its incomplete. Maybe the error log contains the full message or you can
the second one "SQLException: No Suitable driver found for mysql" clearly indicates a classpath issue.
usually connection pools like c3p0 or BoneCP recommed to use a validation query to determine if a connection is valid (something like "select 1" for mysql). That may help to make sure the connection is ok and not rely on the driver?