I am building a web app with a single (not pooled) full time (jdbc) connection between static classes and the database. This is expected to be a low traffic site and the static methods are synchronized. In an effort to speed up access to product information, I am trying PreparedStatements for the first time. As I test, on localhost, sure that I'm the only one running the app., it seems clear to me that my the prepared statements are slower than the unprepared statements I use earlier in the process. This may not be a fair comparison. The unprepared statements get a single result set from one table and it's done. As you can see from the code below, what I'm doing with prepared statements involves three tables and multiple queries. But since this is my first time, I would appreciate review and comment. This does actually work; i.e. all data is retrieved as expected.
The first method below (initialize()) is called once from a servlet init() method when the application is first started. The second method (getItemBatch()) retrieves information about as many product items as match a product name (Titel). My little development / test database has less than 100 (total) items and only 1-3 items matching each name; most often only 1. The server app and database are on the same machine and I'm accessing from a browser via localhost. I'm surprised by the consistent perceptible wait for this detailed product data compared to fetching a master product list (all items) mentioned above.
public static boolean initialize (Connection connArg, String dbNameArg, String dbmsArg) {
con = connArg;
dbName = dbNameArg;
dbms = dbmsArg;
sqlserver_con = connArg;
ItemBatchStatement =
"select Cartnr, Hgrupp, Prisgrupp, Moms from dbo.Centralartregister " +
"where Titel = ?";
ForArtNrStatement =
"select Artnr, Antal from dbo.Artikelregister " +
"where Cartnr = ? and Antal > 0";
ItemHgruppStatement =
"select Namn from dbo.Huvudgrupper " +
"where Hgrupp = ?";
try {
con.setAutoCommit(false);
getItemBatch = con.prepareStatement(ItemBatchStatement);
getForArtNr = con.prepareStatement(ForArtNrStatement);
getItemHgrupp = con.prepareStatement(ItemHgruppStatement);
} catch (SQLException e) {
return(false);
} finally {
try {con.setAutoCommit(true);} catch (SQLException e) {}
}
return(true);
}
-
public static synchronized String getItemBatch (String Titel) throws SQLException {
String ret_xml;
ResultSet rs = null;
ResultSet rs1 = null;
ResultSet rs2 = null;
Titel = charChange(Titel);
ret_xml = "<ItemBatch Titel='" + Titel + "'>";
try {
con.setAutoCommit(false);
getItemBatch.setString(1,Titel);
rs = getItemBatch.executeQuery();
while (rs.next()) {
getForArtNr.setInt(1,rs.getInt("Cartnr"));
rs1 = getForArtNr.executeQuery();
getItemHgrupp.setInt(1,rs.getInt("Hgrupp"));
rs2 = getItemHgrupp.executeQuery();
if (rs1.next() && rs2.next()) {
ret_xml += "<item><Hgrupp>" + rs2.getString("Namn") + "</Hgrupp><Price>" + rs.getInt("Prisgrupp") + "</Price><Moms>" + rs.getInt("Moms") + "</Moms><Cartnr>" + rs.getInt("Cartnr") + "</Cartnr><Artnr>" + rs1.getInt("Artnr") + "</Artnr></item>";
}
}
ret_xml += "</ItemBatch>";
return(ret_xml);
} catch (SQLException e) {
return("SQLException: " + e);
} finally {
try {con.setAutoCommit(true);} catch (SQLException e) {}
if (rs != null) {rs.close();}
if (rs1 != null) {rs1.close();}
if (rs2 != null) {rs2.close();}
}
}
.
UPDATE: I'm still googling and looking for ways to do better. Let me add something for your consideration.
I'm closing the prepared statements via the servlet's destroy() method; the same servlet that calls the method "initialize()" above in order to create them. The idea is to create them once (and only once) and have them available forever for all users (i.e. until the app or server is shut down). I'm going for a kind-of a pseudo-stored procedure. I would just use stored procedures straight out, but the database exists to support another application (their in-house sales system) and I'm going to use it for Internet sales read-only ... avoiding any potential conflict with their maintenance efforts or agreements etc. by not doing anything to change their database set-up. I have suggested that my app use a new account limited to read-only privileges. And come to think of it, I haven't tested whether I can use Prepared Statements in that mode; just seems like it should work.
Each result set is closed in the finally block in the method where they are created. I guess that's ok? I reuse the same RS variable name for multiple result sets (on the second two) but close only once. But wait! Do I even need that? The ResultSets are declared within the scope of the method. If resetting them without closing the old ones doesn't cause leaks, then exiting the method should do just as well on its own. It is a static method, but the ResultSets will be reset every time it's used (at the very least). So, at most, there would just be a single set of ResultSet handles available for reuse; not run-away "leakage."
I'm wondering if I can send the two select requests in the loop both at the same time, simply by turning them into one prepared statement separated by ';'. Or just found "MultipleActiveResultSets=True"; document allowing SQL Server to process multiple transaction requests on a single connection ... still investigating this. Or is there another way to create a prepared statement that would fetch ALL the data via a single submission? (Seems to me like there are too many round trips.) Finally, I might get a boost from using connection pooling, which I haven't done yet. It's low priority in my project right now, but I might have to do it before we go online.
If you create web applicaton and use some web container like tomcat, jetty etc you always can use jdbc datasource and connection pool. It is simple. Good explanation gives here
If you don't want to use connection pool I suppose it would be better use Method Scope Connections described link above
Related
try{
Statement stm = conn.createStatement();
String sql = "SELECT * from BOOKS WHERE ISBN_No = '" + line + "'";
ResultSet rs = stm.executeQuery(sql);
if(//values are returned) {
displayBookInfo(line);
}
else (//if it is null) {
System.out.println("No book found");
}
stm.close();
} catch (SQLException e) {
e.printStackTrace();
System.out.println("Fail to search the book" + line );
noException = false;
}
After I execute the ResultSet rs = stm.executeQuery(sql); I want to check if the query returned a value or if it was empty so that I can execute either "display book details" or "no book found message". I am just confused about how I should compare and how comparison works.
This code is a security leak. You must fix this first.
You cannot include untrusted inputs in a query like this. What if someone enters, say:
1234'; DROP TABLE books CASCADE; EXECUTE 'FORMAT C: /Y'; --
In the web form? Don't try it, you'll wipe your disk. You get the point, surely.
The right way is to use stm.prepareStatement(sql), where sql is a constant (so not something you insert user entered stuff into), using a ? where user input is needed, then calling .setString(1, line) to then tell your db driver what should go in place of the question mark.
Then, simply rs.next(), which advanced to the next row in the result (first call advances to the first row). If there are no rows left, it returns false instead. Hence, if your query returns 0 rows, the first resultSet.next() call returns false right away.
Your code also fails to close. You must use try-with-resources on everything (ResultSet, (Prepared)Statement, and most importantly the Connection), or your app will crash after a few statements.
NB: Minor nit, if all you want to know is if there's at least one result, add LIMIT 1, and just SELECT 1 FROM instead - it's less overhead that way.
I'm trying to move data from one table to another(both table are same basically), the method that I'm using is INSERT-SELECT.
The problem that I'm facing is my java program seem like frozen there, but I can still force close it with ^C easily, so I think it might be still alive but stuck for some reason.
This is my code which stuck in this problem
public String moveData(String sql, int day) {
Connection con = null;
PreparedStatement stmt = null;
int count;
try {
con = DriverManager.getConnection(DSN, Username, Password);
stmt = con.prepareStatement(sql);
stmt.setInt(1, day);
count = stmt.executeUpdate();
return String.valueOf(count);
} catch (SQLException e) {
System.out.println("SQL exception: " + e.getMessage());
e.printStackTrace();
return "false";
} finally {
try {
stmt.close();
} catch (Exception e) {
}
try {
con.close();
} catch (Exception e) {
}
}
}
and the SQL, executing it in SQL developer is really fast with 0.231 sec for 23k row of data
insert into Request_History (customer_id, request_id, status, transaction_date, last_modified)
SELECT customer_id, request_id, status, transaction_date, current_timestamp
from Request_Log
where transaction_date <= (sysdate - NUMTODSINTERVAL(:1 ,'DAY'))
I see no problem on them, is there anything that I missed?
Update
Since there's no resolve on the program and SQL command, I'd like to change a way of thinking on the DB side.
Could anyone please tell me what kind of privileges do I need to execute my INSERT-SELECT SQL command on 11g without problem? because from what I can see that this command would only needs basic privileges such as SELECT/INSERT/UPDATE/DELETE to execute.
The time of the whole insert process depends on three factors:
How many rows are in the query result set
Is there any index, which can be used at query level (best for this would be an index on Request_Log.transaction_date
Are there any constraints, indexes which have to be maintained during the insert phase (Request_History table)
Check the exec plan of the statement, to see if there's anything - worst things are full table scans. If you allowed, you can paste here the execution plan as well.
there is probably something wrong with the way you open your connection, something with you session parameters. You might not have enough undo space or another limiting factor. Maybe you can get a dba to check on it while you execute this over java vs. SQL Developer.
alter you statement to
create table Request_History_test
as SELECT
customer_id, request_id, status, transaction_date, current_timestamp
from
Request_Log
where
transaction_date <= (sysdate - NUMTODSINTERVAL(:1 ,'DAY'))
if this goes fast in contrast to your "insert select", then contact your DBA. There might be known issues with your undo space / redo logs. I'm not a DBA, so this answer is vague, I just had similar problems once.
oh, and check your dba_hist_sqlstat to see where the time is lost.
I know this probably is a similar question from the rest, (well originally, before I tried something new, it was a bit unique but it never solved the main problem), but I probably need to discuss this with someone who can help because I could never get what's causing this despite already reading various posts from this site. Bottom line is I need to keep on making plenty of sequential queries but I ended up making too many connections.
What my program does is that it displays data about each member and that it's sort of a tree or network where, in order to get the data you need for each member, you have to scout through every other member that points to that current member (or child's data) , and the data of the member that points to the member that points to the current member (or grandchild's data) and so on. Hence, why I need to keep making queries cause I need to get the data off of each child. Each node has I think a minimum children of 5 and on my 34th member, it gave off that "Too Many Connections" error.
I have read how to open and close the Connections and all but am I still doing it incorrectly? I've tried changing the max connections but that's not really a long term solution for me. Here's how I do it:
public class SQLConnect {
private Connection con;
private Statement st;
private ResultSet rs;
public SQLConnect() {
try {
Class.forName("com.mysql.jdbc.Driver");
con = DriverManager.getConnection("jdbc:mysql://localhost:3306/dbname?zeroDateTimeBehavior=convertToNull", "root", "");
st = con.createStatement();
} catch (ClassNotFoundException | SQLException ex) {
System.out.println("Error in constructor: " + ex);
}
}
//this method gets called before I make another query
public void reconnect() {
try {
st.close();
con.close();
if (con.isClosed()) {
con = DriverManager.getConnection("jdbc:mysql://localhost:3306/dbname", "root", "");
st = con.createStatement();
}
} catch (SQLException ex) {
Logger.getLogger(SQLConnect.class.getName()).log(Level.SEVERE, null, ex);
}
}
//sample method on how I do queries
public ResultSet getMemberViaMemberId(String mID) {
try {
String query = "CALL getMemberViaMemberId(" + mID + ");"; //procedure call
rs = st.executeQuery(query);
} catch (Exception ex) {
System.out.println("Error: " + ex);
}
return rs;
}
}//end of class
The way I call it in my JForm is this..
SQLConnect connect;
public Class(){
connect = new SQLConnect();
}
public void methodThatGetsCalledALot(String current_id){
connect.reconnect(); //refer to SQLConnectClass displayed above
ResultSet member = connect.getMemberViaMemberId(current_id);
try{
if (member.next()) {
lastName = member.getString("last_name");
firstName = member.getString("first_name");
}
//display data...
} catch (SQLException ex){
}
}
The code:
connect.reconnect();
ResultSet rs = connect.callSQLMethod();
is the most essential bit and is called by every class, and by every method that needs to fetch data. I have to acknowledge that I never bother closing ResultSet because often times it's inside a loop and gets replaced with new data anyway.
Again, my problem is: I cant continue fetching data anymore because of too many connections. Am I really closing things properly or am I missing something? Any suggestions on how to fix this? If my question is too confusing, I'd add more details if required. Thank you. If anyone's to keen on freely helping me out, I'd go for some emailing. Thank you! And Happy New Year btw.
You seem to be creating a lot of connections and recursing with the ResultSet open. Don't create new connections all the time, all you need is one connection and don't reconnect all the time. You actually don't need the reconnect method at all (unless you connection closes automatically, in which case you can check if it is closed before executing query). And you need to close the ResultSet once you are done retrieving values.
All you need is the data and not the resultset. So take the data and release the resource ie ResultSet. So do this -
In your getMemberViaMemberId don't return ResultSet, in that method itself, iterate through the resultset and create the object for the row and store it into a collection and return that collection after closing the ResultSet. And dont call reconnect method at all.
Close the single connection that you have when exiting the program.
Code:
// Start the query.
final ResultSet r = prepared.executeQuery();
try {
// Returning false from oneRow will stop the process.
boolean goOn = true;
while (r.next() && goOn) {
log.trace("Result: {}", toString(r));
// We did do at least one.
ran = true;
goOn = oneRow(r);
}
} finally {
try {
// Always remember to close the ResultSet.
r.close();
} catch (SQLException ex) {
log.error("Close failed", ex);
}
}
// Handle one row.
public boolean oneRow(ResultSet r) throws Exception {
String xml1 = r.getString(1);
String xml2 = r.getString(2);
if (xml1 == null && xml2 == null) {
// Probably compressed.
xml1 = decompress(r, 3);
xml2 = decompress(r, 4);
}
return false;
}
private static String decompress(ResultSet rs, int col) throws SQLException {
// Exception gets thrown here!!! ???
final InputStream compressed = rs.getBinaryStream(col);
...
As you can see this is not the obvious problem of not calling ResultSet.next(). Not only that but I have called getString in the ResultSet twice already, it is the third time that breaks it.
Note that this is an occasional issue, this code works fine most of the time.
The query looks something like:
"SELECT P1.XML XML1, "
+ "P2.XML XML2, "
+ "P1.CompressedXML CompressedXML1, "
+ "P2.CompressedXML CompressedXML2 "
+ "FROM Table1 P1 "
+ "LEFT JOIN Table2 T2 ON T2.ID = P1.ID "
+ "LEFT JOIN Table1 P2 ON P2.ID = T2.Item_Code "
+ "WHERE P1.ID = ?"
I realise this is a rather incestuous query but as I stated - this all works fine most of the time.
Added
Prompted by the answer posted by user1933888 it occurs to me that since I am using a home-grown connection-pool is it possible for a prepared statement to interfere with itself between two threads sharing different connections?
I am confident that the same connection will never be in use by two threads at the same time but the prepared statement could be shared as it should reside in the database.
As your PreparedStatement is linked to a connection, if you share the PreparedStatement between threads you would end up sharing a connection and that definitely will cause problems.
It is true that statements are generally cached in the DB to avoid having to reparse it but you'd still need one PreparedStatement object per thread to avoid confusing the DB driver as to what thread wants what.
In general, I would avoid a home brew connection pool considering that in general AppServers do the pooling for you or (if you're not in an AppServer) there are plenty of connection pool libs out there, so why reinvent the wheel.
In general:
you can shared DataSources between concurrent threads
you should NOT share Connection, PreparedStatement or ResultSet objects between concurrent threads
The connection pool will ensure that non-concurrent threads can reuse Connections and the JDBC driver will internally cache Statements
Hope that makes sense.
I am working on a project for uni (happens to be due in 14 hours) and I am at a sticking point. It is a web based web store running in eclipse on apache tomcat and derby.
I have a prepared statement that checks for a user name and passwordhash, no matter what I try this statement returns 0 rows. The same sql runs in the sql scratch pad and returns what is expected.
I have used the debugger to inspect the prepared statement object and the query seems fine. The ?'s in the text are still in place rather than filled with the variables, but that seems normal. I have also tried to run the exact same hand written sql from the console, but without any luck.
The query I run in the sql console is
SELECT * FROM username WHERE username='user#system.com' AND passwordhash='passwordhash'
The prepared statments look like this.
PreparedStatement pstmt = db.prepareStatement("SELECT * FROM reallynice.username " +
"WHERE emailaddress=?" +
" AND passwordhash=?");
pstmt.setString(1,username);
pstmt.setString(2, username + ":" + passwordLogin);
I am at the point where I have tried everything, and have run out of searches to make. I know this is a uni project and the standard reply is to give people somewhere to look. At this point I need spoon feed a path to go down.
EDIT Here is some more background, I have tried running a known working query in this pipeline and it also fails to return any rows.
public static User getUser(String username, String passwordHash) {
DBBean db = new DBBean();
System.out.println("Logging in for username " + username + " and password " + passwordHash);
try {
ResultSet rs;
PreparedStatement pstmt = db.prepareStatement("SELECT * FROM reallynice.username " +
"WHERE emailaddress=?" +
" AND passwordhash=?");
pstmt.setString(1,username);
pstmt.setString(2,passwordHash);
//PreparedStatement pstmt = db.prepareStatement("SELECT * FROM reallynice.product");
//PreparedStatement pstmt = db.prepareStatement("SELECT * FROM reallynice.username WHERE emailaddress='user#me.com' AND passwordhash='megahashstring'");
rs = pstmt.executeQuery();
System.out.println("Rows returned\t" + rs.getRow());
if(rs.getRow() < 1)
return null;
int id = rs.getInt("uid");
String name = rs.getString("name");
String emailaddress = rs.getString("emailaddress");
String password = rs.getString("passwordhash");
boolean isAdmin = false;
pstmt = db.prepareStatement("SELECT * FROM reallnice.admin WHERE uid= ?");
pstmt.setInt(1, id);
rs = pstmt.executeQuery();
if(rs.getMetaData().getColumnCount() > 0)
isAdmin = true;
return new User(id,isAdmin,name,emailaddress,password);
} catch(Exception ex) {
System.out.println(ex);
}
return null;
}
I have also included the other queries I have tried for this.
Whenever I see someone having an experience like this: "no matter what I try this statement returns 0 rows," there are two possible reasons that come immediately to mind:
1) You aren't using the database you think you are. Derby's connection URL, if you say ";create=true", will quite happily make a new, empty database when you connect, if it doesn't find an existing database in the location you expect. This sort of problem arises from a confusion over where the databases are created; a database with a relative name will be created in whatever directory turns out to the be derby.system.home of the Derby instance that gets that connection URL. So check to see if you are using a different current working directory, or for some other reason are connecting to a different database than you think you are.
2) You aren't using the schema you think you are. Derby will quite happily create multiple schemas, and each schema has a separate set of tables, so if you are initially connecting as user A, and then later connect as user B, and don't issue SET SCHEMA, then user A and user B have completely separate sets of tables and so you won't be accessing the tables that you think you are. So check to see if you are connecting as the same user and using the same schema when you connect to the database.
Try changing how you display your logging statement
System.out.println("Rows returned\t" + rs.getRow());
getRow() returns the current row number, not how many records were returned. In order to user getRow() to count the number of entries in the result set you would need to move the pointer of the result set to the last entry.
You have also, not called next() yet, which means you aren't pointing at anything (and most likely the reason you always see 0 as the number). Try using
while(rs.next()){ //go through the entire ResultSet}
or
if(rs.next()) { //access the first record in the ResultSet}
So over all, if you change your code to something like the following you may have better results.
rs = pstmt.executeQuery();
if(rs.next()){
System.out.println("Processing Row " + rs.getRow());
//continue on
}else{
System.out.println("No Records");
}
If you have set your table where the username is a unique key, you can be assured this will return 0 or 1 row. Otherwise use the while() option instead of if()
EDIT::
Also as a side note, because you are not calling next()
if(rs.getRow() < 1)
return null;
will always be 0, which returns null from your method.