I am trying to execute a query using postgre sql driver for java jdbc.
I have an issue with memory buildup my statement is in a loop and then sleeps.
The problem is when I look at the job in task manager I can see the memory climbing 00,004K at a time. I have read the documentation I have closed all connections statements resultsets but this still happens.
Please could you tell me what is causing this in my code.
String sSqlString = new String("SELECT * FROM stk.comms_data_sent_recv " +
"WHERE msgtype ='RECEIVE' AND msgstat ='UNPRC' " +
"ORDER BY p6_id,msgoccid " +
"ASC; ");
ResultSet rs = null;
Class.forName("org.postgresql.Driver");
Connection connection = DriverManager.getConnection(
"jdbc:postgresql://p6tstc01:5432/DEVC_StockList?autoCloseUnclosedStatements=true&logUnclosedConnections=true&preparedStatementCacheQueries=0&ApplicationName=P6Shunter", "P6dev",
"admin123");
//Main Loop
while(true)
{
try{
Statement statement = connection.createStatement();
statement.executeQuery(sSqlString);
//rs.close();
statement.close();
//connection.close();
rs = null;
//connection = null;
statement =null;
}
finally {
//connection.close();
}
try {
Thread.sleep(loopTime);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Notice the commented out code.. I did close all but that did not seem to make a difference. Whet I did see is that it seems that the statement executeQuery(sSqlString); is causing this the reason I think so is if I remove the statement there is no memory leak.
I could be wrong but please assist me.
UPDATE:
I have changed my code as with your recommendations. Hope its a bit better please let me know if I need to change something.
My main loop :
public static void main(String[] args) throws Exception {
// TODO Auto-generated method stub
//Main Loop
while(true)
{
getAndProcessAllUnprcMessagesFromStockList();
try {
Thread.sleep(loopTime);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
My Function it will call do fetch data :
public static void getAndProcessAllUnprcMessagesFromStockList() throws Exception
{
ResultSet rs = null;
Statement statement = null;
Connection connection =null;
String sSqlString = new String("SELECT * FROM stk.comms_data_sent_recv " +
"WHERE msgtype ='RECEIVE' AND msgstat ='UNPRC' " +
"ORDER BY p6_id,msgoccid " +
"ASC; ");
try{
Class.forName("org.postgresql.Driver");
connection = DriverManager.getConnection(
"jdbc:postgresql://p6tstc01:5432/DEVC_StockList?autoCloseUnclosedStatements=true&logUnclosedConnections=true&preparedStatementCacheQueries=0&ApplicationName=P6Shunter", "P6dev",
"admin123");
PreparedStatement s = connection.prepareStatement(sSqlString,
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
rs = s.executeQuery();
while (rs.next()) {
//Process records
UnprcMsg msg = new UnprcMsg();
msg.setP6Id(rs.getString(1));
msg.setMsgOccId(rs.getString(2));
msg.setWsc(rs.getString(3));
msg.setMsgId(rs.getString(4));
msg.setMsgType(rs.getString(5));
msg.setMsgStatus(rs.getString(6));
//JOptionPane.showMessageDialog(null,msg.getP6Id(), "InfoBox: " + "StockListShunter", JOptionPane.INFORMATION_MESSAGE);
//msg2 = null;
}
rs.close();
s.close();
}
catch(Exception e)
{
e.printStackTrace();
}
finally
{
connection.close();
}
}
I have closed my connections statements and results.
I also downloaded eclipse memory analyzer and I ran the jar witch will execute my main loop. Ran it for about an hour and here's some of the data I got from memory analyzer..
Leak suspects :
Now I know I cant go on the memory usage of task manager but whats the difference? Why does task manager show the following :
I was concerned about the memory usage I see in task manager? should I be?
I have the code that successfully establishes a connection to a mySQL database.
String email, password; //assume these are already loaded with user-entered data.
try {
Class.forName("com.mysql.jdbc.Driver");
} catch (ClassNotFoundException e) {
return false;
}
Connection conn = null;
try {
conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/main", "root", "password123");
} catch (SQLException e) {
return false;
}
//perform my database actions here///////////////
///////////////////////////////////////////////////
try {
conn.close();
} catch (SQLException e) {
return false;
}
I have a couple of strings in the scope of the code above that already have the email and password entered by a user on a login page. I need to look through the database for a matching email address and then verify that the password matches what the user entered in the form.
My table has 3 columns: id, email, and password.
I have pushed two rows into the table using the sql workbench
1 | email#gmail.com | password1
2 | email2#gmail.com | password2
I'm assuming in pure SQL I have to do something like
SELECT * FROM users WHERE email LIKE 'email#gmail.com' AND password LIKE 'password1';
But I'm not quite sure how to actually send these SQL commands to the database and receive info back using JSP. Also, I'm not entirely sure my SQL logic is the ideal way to verify a password. My thinking with the SQL command above was that if the database finds any row that meets the conditions, then the email/password combination are verified. Not sure if this is a great way to do it though. I'm not looking for the most secure and complicated way, I'm just looking for the simplest way that makes sense at the moment. Every tutorial I find seems to do it differently and I'm a bit confused.
Here's an example you can use from something I've worked on (I'm assuming that the connection "conn" is obvious):
PreparedStatement st = null;
ResultSet rec = null;
SprayJobItem item = null;
try {
st = conn.prepareStatement("select * from sprayjob where headerref=? and jobname=?");
st.setString(1, request.getParameter("joblistref"));
st.setString(2, request.getParameter("jobname"));
rec = st.executeQuery();
if (rec.next()) {
item = new SprayJobItem(rec);
}
} catch (SQLException ex) {
// handle any errors
ReportError.errorReport("SQLException: " + ex.getMessage());
ReportError.errorReport("SQLState: " + ex.getSQLState());
ReportError.errorReport("VendorError: " + ex.getErrorCode());
} catch (Exception ex) {
ReportError.errorReport("Error: " + ex.getMessage());
} finally {
// Always make sure result sets and statements are closed,
if (ps != null) {
try {
ps.close();
} catch (SQLException e) {
;
}
ps = null;
}
if (rs != null) {
try {
rs.close();
} catch (SQLException e) {
;
}
rs = null;
}
}
In your case instead of item = new SprayJobItem(rec);
you would have code that notes that the user is valid as the record has been found.
I have the following code, which does not get to the .setText("Successful") statement, indicating an issue with the drivermanager.getConnection statemenet (I think). It finds the database driver that I'm using from net.sourceforge. But there is no exception error message thrown, nothing happens:
String connectionurl = "jdbc:jtds:sqlserver://41.185.13.201; databaseName=Courses; user=*;Password=*;Persist Security Info=True;";
try {
Class.forName("net.sourceforge.jtds.jdbc.Driver").newInstance();
Connection con = DriverManager.getConnection(connectionurl);
textview7.setText("Successful");
// Create and execute an SQL statement that returns some data.
String SQL = "INSERT INTO Courses (CourseCode) VALUES ('INFO3002')";
Statement stmt = con.createStatement();
stmt.executeUpdate(SQL);
con.close();
}
catch (ClassNotFoundException e) {
textview7.setText("Could not find the database driver " + e.getMessage());
} catch (SQLException e) {
textview7.setText("Could not connect to the database " + e.getMessage());
}
catch (Exception e) {
//textview7.setText(e.getMessage());
}
You should access your data through web services (JAX-WS or JAX-RS). It is the best architecture you can use. As said above, you should simply develop a middleware.
I am trying to use an SQL database with a Java program. I make a table that is 7 columns wide and 2.5 million rows (My next one I need to build will be about 200 million rows). I have two problems: building the SQL table is too slow (about 2,000 rows/minute) and searching the database is too slow (I need to find over 100 million rows in under a second if possible, it currently takes over a minute). I have tried creating a csv file and importing it, but I can't get it to work.
I am using xampp and phpMyAdmin on my computer (i5 + 6gb ram). I have three methods I am testing: createTable(), writeSQL(), and searchSQL().
createTable:
public static void createTable() {
String driverName = "org.gjt.mm.mysql.Driver";
Connection connection = null;
try {
Class.forName(driverName);
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
String serverName = "localhost";
String mydatabase = "PokerRanks4";
String url = "jdbc:mysql://" + serverName + "/" + mydatabase;
String username = "root";
String password = "";
try {
connection = DriverManager.getConnection(url, username, password);
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
///////////////
String table = "CREATE TABLE ranks(deckForm bigint(10) NOT NULL,rank0 int(2) NOT NULL,rank1 int(2) NOT NULL,rank2 int(2) NOT NULL,rank3 int(2) NOT NULL,rank4 int(2) NOT NULL,rank5 int(2) NOT NULL,PRIMARY KEY (deckForm),UNIQUE id (deckForm),KEY id_2 (deckForm))";
try {
Statement st = connection.createStatement();
st.executeUpdate(table);
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
///////////////
try {
connection.close();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
writeSQL():
public static void writeSQL() {
String driverName = "org.gjt.mm.mysql.Driver";
Connection connection = null;
try {
Class.forName(driverName);
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
String serverName = "localhost";
String mydatabase = "PokerRanks4";
String url = "jdbc:mysql://" + serverName + "/" + mydatabase;
String username = "root";
String password = "";
try {
connection = DriverManager.getConnection(url, username, password);
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
/////////////// Prepared Statement with Batch
PreparedStatement statement = null;
String sql = "INSERT INTO ranks VALUES (? ,0, 0, 0, 0, 0, 0)";
long start = System.currentTimeMillis();
try {
statement = connection.prepareStatement(sql);
for (int i = 0; i < 100; i++) {
for (int j = 0; j < 100; j++) {
statement.setLong(1, (i*100 + j));
statement.addBatch();
}
System.out.println(i);
statement.executeBatch();
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (statement != null) {
try {
statement.close();
} catch (SQLException e) {
} // nothing we can do
}
if (connection != null) {
try {
connection.close();
} catch (SQLException e) {
} // nothing we can do
}
}
System.out.println("Total Time: " + (System.currentTimeMillis() - start) / 1000 );
///////////////
}
searchSQL():
public static void searchSQL() {
String driverName = "org.gjt.mm.mysql.Driver";
Connection connection = null;
try {
Class.forName(driverName);
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
String serverName = "localhost";
String mydatabase = "PokerRanks2";
String url = "jdbc:mysql://" + serverName + "/" + mydatabase;
String username = "root";
String password = "";
try {
connection = DriverManager.getConnection(url, username, password);
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
/////////////// Option 1, Prepared Statement
ResultSet rs = null;
PreparedStatement pstmt = null;
String query = "SELECT rank0, rank1, rank2, rank3, rank4, rank5 FROM ranks WHERE deckForm = ?";
long start = System.currentTimeMillis();
try {
pstmt = connection.prepareStatement(query);
for (int i = 0; i < 100000; i++) {
pstmt.setLong(1, 1423354957);
rs = pstmt.executeQuery();
while (rs.next()) {
int[] arr = {rs.getInt(1), rs.getInt(2), rs.getInt(3), rs.getInt(4), rs.getInt(5), rs.getInt(6)};
}
}
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println("Total Time: " + (System.currentTimeMillis() - start) / 1000 );
///////////////
/*
/////////////// Option 2
Statement st = null;
long start = System.currentTimeMillis();
try {
st = connection.createStatement();
ResultSet rs = null;
long deckForm = 1012213456;
for (int i = 0; i < 100000; i++) {
rs = st.executeQuery("SELECT rank0, rank1, rank2, rank3, rank4, rank5 FROM ranks WHERE deckForm = " + deckForm);
while (rs.next()) {
int[] arr = {rs.getInt(1), rs.getInt(2), rs.getInt(3), rs.getInt(4), rs.getInt(5), rs.getInt(6)};
}
}
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println("Total Time: " + (System.currentTimeMillis() - start) / 1000 );
///////////////
*/
try {
connection.close();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Sorry that's so long. I've tried everything I can think of to make this faster but I can't figure it out. Any suggestions?
Well, there's a few improvements you could make:
You are creating a connection each time you want to search, write or create,
you should use a pooled connection and datasources.
Optimize your queries by doing explain plans, and optimize your table relations and indexes.
You can use stored procedures and call them.
Well that's all I can help with, certainly there are more tips.
As to the insert speed, you need to disable all the indexes prior to doing the insert and re-enable them after you're done. Please see Speed of Insert Statements for a lot of detailed information on improving bulk insert speed.
The query speed is probably limited by your CPU and disk speeds. You may have to throw much more hardware at the problem.
building the SQL table is too slow (about 2,000 rows/minute)
So point of view on inserting a great number of rows is sure use Heap table, it's basic table, also it named as persistent page-array usually created just by CREATE TABLE, it's not effective for searching as you meant that search is slow but for inserting is very efficient because it add rows to first free position that what find or on the end of table. But on other hand, searching is very inefficietly, because is not guaranteed sort of items/rows.
searching the database is too slow (I need to find over 100 million
rows in under a second if possible, it currently takes over a minute)
So for this you should create table in that is searching is efficiently. In a case if you using Oracle, so it offers many constructions for physical implementation for example Index organized tables, Data clustering, Clustered tables - Index / Hash / Sorted hash ...
SQL Server i'm not sure but also clustered tables and MySQL i don't know exactly, i don't want to tell you something worst. I don't say that MySQL is bad or worse like Oracle for example but just not offer some techniques for physical implementation like Oracle for example
So, i mean that it's quite hard to say some recommendations for this approach but you seriously think and study something about physical implementations of database systems, have look at relational algebra for optimize your statements, which types of tables you should create, #duffymo meant right that you can let explain your query execution plan by EXPLAIN PLANE FOR and based on result to optimize. Also how to use indexes, it's strong database construction but each index mean much more operations for any modifying of database so well to rethink for which attribute you create index etc.
Via Google, you find many useful articles about data modeling, physical implementation etc.
Regards man, I wish best of luck
I'm trying to hit a stored procedure but I'm getting this error message: 'javax.ejb.EJBException'... I've never worked with stored procedures so the exception is a bit Greek to me.
Anyone that could perhaps shed some light on this? Below I pasted the code that I wrote:
#WebMethod(operationName = "getSpecimenResultsXml")
public String getSpecimenResultsXml(#WebParam(name = "specimenGuid") String specimenGuid, #WebParam(name = "publicationGuid") String publicationGuid, #WebParam(name = "forProvider") String forProvider) {
//Method variables
ResultSet rs = null;
String xml = null;
// 1) get server connection
Connection conn = dataBaseConnection.getConnection();
// 2) Pass recieved parameters to stored proc.
try {
CallableStatement proc =
conn.prepareCall("{ call getSpecimenReportXml(?, ?, ?) }");
proc.setString(1, specimenGuid);
proc.setString(2, publicationGuid);
proc.setString(3, forProvider);
proc.execute();
rs = proc.getResultSet();
} catch (SQLException e) {
System.out.println("--------------Error in getSpecimenResultsXml------------");
System.out.println("Cannot call stored proc: " + e);
System.out.println("--------------------------------------------------------");
}
// 3) Get String from result set
try {
xml = rs.getString(1);
} catch (SQLException e) {
System.out.println("--------------Error in getSpecimenResultsXml------------");
System.out.println("Cannot retrieve result set: " + e);
System.out.println("--------------------------------------------------------");
}
// 4) close connection
try {
conn.close();
} catch (Exception e) {
System.out.println("--------------Error in getSpecimenResultsXml------------");
System.out.println("Cannot close connection: " + e);
System.out.println("--------------------------------------------------------");
}
// 5) return the returned String
return xml;
}
Oh and the stored procedure us called getSpecimenReportXml...
Your exception would say 'caused by' somewhere - which is a big clue. If it's an NPE then you might want to check the values of dataBaseConnection and conn to make sure they've been set. Use a debugger to do this, but the exception should tell you exactly which line caused the problem.