My objective is this:
I have a SELECT query with that I want to run on a database with a lot of rows. This query will produce a LOT of results, so I was thinking of running it in iterations with LIMIT 1000 and OFFSET ? where ? will be the last row that were processed in each iteration.
Eg:
Get rows 1 - 1000
Get rows 1001 - 2000
Get rows 2001 - 3000
Get rows 3001 - 4000
...
I was thinking about doing it in a loop where the each new iteration will set the last iteration's row as the new iteration's OFFSET (Eg: OFFSET 1001, OFFSET 2001, etc. as shown above).
I am new to using JDBC, so is this the correct way to do it? If so, how do I re-use PreparedStatement when I must execute it and get the result for each iteration?
If this isn't the correct way to do it, what is the correct way?
EDIT:
Here's my current code:
private static void import(Date from, Date to) throws Exception {
PreparedStatement p = connect.prepareStatement(statement);
p.setInt(1, 0);
p.add
ResultSet results;
for (int i=0; i< WAVES; i++) {
results = p.executeQuery();
Integer lastRow = importFrom(results);
p.setInt(1, lastRow.intValue()+1);
results.close();
}
p.close();
}
EDIT 2:
Here's the SQL String:
SELECT
item.aitem_id, item.action_type, item.user_id, item.pid, item.pr_id, item.action_time, item.notes, item.screen, item.vid, item.lo_id,
vals.value_name, vals.simple_int_value, vals.simple_double_value, vals.simple_date_value, vals.simple_string_value,
data.version_id, data.prev_version_id
FROM mySchema.aitems item
JOIN mySchema.avalues vals ON item.aitem_id=vals.aitem_id
JOIN mySchema.adata data ON item.aitem_id=data.aitem_id
LIMIT 1000 OFFSET ?;
I modified both my code and SQL to prepare with only the OFFSET.
What your doing is correct but It would be a good idea to add a call to clearParameters() and uses try finally blocks. Below is how I would implement it
PreparedStatement p = null;
ResultSet results = null;
Integer lastRow = 0;
try
{
p = connect.prepareStatement(statement);
p.add
for (int i=0; i< WAVES; i++)
{
p.clearParameters();
p.setInt(1,lastRow.intValue() + 1)
try // This try might not really be necessary
{
results = p.executeQuery();
Integer lastRow = importFrom(results);
p.setInt(1, lastRow.intValue()+1);
} // Add dealing with exceptions
finally
{
results.close();
}
}
} //Add dealing with exceptions
finally
{
p.close();
}
Related
I created a system in which I can run all my postgre sql queries with only one single Async Task Class in Android Studio. This was really(!!) challenging due to the big amount of limitations that I had to face. But this works actually really great!
//Used for connecting to database and executing queries.
//Index 0 of input string must be the query, Index 1 must be the tablename we demand
//We can only gather data from 1 table for each query, so if you need data from several tablecolumns, use multiple queries like:
//[0] = query, [1] = tablename, [2] = 2nd query, [3] = 2nd tablename, [4] = 3rd query, [5] = 3rd table name ... and so on (each query must come with a tablename)
public class DBHandler extends AsyncTask<String, Void, List<String>>
{
public AsyncResponse delegate;
#Override
protected List<String> doInBackground(String...query)
{
List<String> result = new ArrayList<String>();
String sql;
String tableresult = null;
Connection conn = null;
Statement st = null;
ResultSet rs = null;
try {
Class.forName("org.postgresql.Driver");
conn = DriverManager.getConnection("jdbc:postgresql://192.168.200.300:5439/dbname?user=anonymous&password=secretpw");
st = conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_UPDATABLE); //necessary if you want to use rs.first() after rs.next(), it makes the resultset scrollable
for (int i = 0; i <= query.length-1; i = i+2) //queries are always stored in i=0 and/or in i+2, because i+1 contain the demanded tablenames for resultset handling
{
System.out.println("I is: " +i);
if (!query[i].isEmpty())
{
System.out.println(query[i]);
sql = query[i];
rs = st.executeQuery(sql);
while (rs.next())
if (!query[i + 1].isEmpty() || !rs.getString(query[i + 1]).isEmpty()) //if i+1 is empty, there is no demanded tablename. Used when we dont need any return values (ie. INSERT, UPDATE)
result.add(rs.getString(query[i + 1])); //demanded tablename is always stored in i+1
//We add an empty entry if we demand multiple tablenames so we can keep them seperate
//Might be replaced with any other char, but you will have to backtrack all usages of DBHandler and fix the filters there
if(i+2 < query.length)
result.add(" ");
}
rs.first(); //reset pointer for rs.next()
}
rs.close();
st.close();
conn.close();
System.out.println("End of AsyncTask");
}
catch (SQLException ex)
{
ex.printStackTrace();
}
catch (Exception e)
{
e.printStackTrace();
}
return result;
}
//onPostExecute returns query result in a List.
//We need to use interaces delegate feature to send the result to other classes, like "Auslieferung", which is implementing the interface
#Override
protected void onPostExecute(List<String> result)
{
super.onPostExecute(result);
System.out.println("Result: " +result.toString());
if (!result.isEmpty())
delegate.processFinish(result);
}
}
There is a for-loop in this Async Task.
for (int i = 0; i <= query.length-1; i = i+2)
And now finally I can explain my issue:
I usually use SELECT queries, sometimes I use an INSERT query (which can be done by a single query), but when I parse an Update Query, my for-loop stops iterating after the first pass, so i+2 never happens. The update queries look like this:
String updatequeries[] = {UPDATE delivery SET contactperson = 'Jon Doe' WHERE officeid = 5, " ", UPDATE delivery SET contactemail = 'abd#def.gh' WHERE officeid = 5, " "};
Why does this for loop stop running right after the first run? The debugger does not show anything unusual, everything was parsed right and there are no queries missing. Updating a table does not return any results, but nothing depends on result values here. I tried to run 20 update queries in a single string var, but the for loop stops after the first iteration anyway. No issues are displayed in the debugger or in the logs. Have I overseen something or is there anything I don't know? Might this be a bug? Please help me! This issue drives me crazy.
Is there an efficient way to obtain a list (preferably an array, a ResultSet will do) to SELECT a lot of rows.
For example:
Connection con = DriverManager.getConnection(host,username,password);
String sql = "SELECT * FROM table_name WHERE food = ? AND expiration > ?";
PreparedStatement stmt = con.prepareStatement(sql);
Using the above code, I want to get all the food from a given array that isn't expired.
String[] foodList = {"banana","apple","orange",...}
where the expiration date is a constant date (lets say 3 days ago). However, the way I have it is that the String and PreparedStatement are in a for loop that loop the number of foods in the array to individually check the expiration date. This creates a lot of ResultSets after I execute each individually.
Most SQL Databases support a IN (list) expression. This is roughly equivalent to providing a or expression:
SELECT id FROM table WHERE food IN ('Apple', 'Banana') AND exp < ?
is similar to
SELECT id FROM table WHERE (food = 'Apple' or food = 'Banana') AND exp < ?
In both cases some RDBMS can optimize it.
However first of all there is a limitation in the number of list items you can specify in the IN or number of characters you can use in the statement. So if your list can be variable long you need to be prepared to run multiple statements.
Secondly you cannot* set a array as an argument to a PreparedStatement and expect it to work with IN.
Unfortunately in plain JDBC all you can do is to concatenate a String. This is frowned upon, but there is no good alternative (unless you want to do something like giving the list of foods as a single list and use a "instring" expression).
Make sure to add as many ? (but not too many) as you expect parameters and then set them in the IN:
String[] foods = ...;
int remain = foods.length;
int start = 0;
while(remain > 0)
{ if (remain >= 100)
executeBatch(foods, start, 100); start+=100; remain-=100;
else if (remain >= 30)
executeBatch(foods, start, 30); start+=30; remain-=30;
else {
executeBatch(foods, start, 1); start+=1; remain-=1;
}
}
void executeBatch(String[] f, int off, int len)
{
StringBuilder sqlBuf = StringBuilder("... IN(");
for(int i=0;i<len;i++) {
sqlBuf.append((i!=0)?",?":"?");
}
String sql = sqlBuf.append(") AND exp < ?").toString();
PreparedStatement ps = c.prepareStatement(sql);
for(int i=0;i<foods.length;i++)
ps.setString(i+1, foods[i+off]);
ps.setTimestamp(foods.length+1, now);
....
}
This avoids to generate a lot of different SQL statement to compile. (Only 100,30 or 1 ?)). You can use the same logic for the OR case.
* not to be confused with ARRAY database type.
Probably not the most elegant solution, and you won't get any performance benefit from the prepared statement (but you will get parameter binding):
StringBuilder sql = new StringBuilder("SELECT * FROM table_name WHERE expiration > ? AND food IN (");
for (int i = 0; i < foodList.length; i++) {
if (i > 0) {
sql.append(',');
}
sql.append('?');
}
sql.append(")");
Connection con = DriverManager.getConnection(host, username, password);
PreparedStatement stmt = con.prepareStatement(sql.toString());
stmt.setDate(1, expirationDate);
for (int i = 0; i < foodList.length; i++) {
stmt.setString(i + 2, foodList[i]);
}
ResultSet rs = stmt.executeQuery();
/* ... Do Stuff ... */
I am writing a program to provide stock data to a GUI, which will update real time. When I use the Smart Tags in Excel, it updates with 13 columns: Last, Previous, Close, High, Low, Volume, Change, % Change, 52 Wk High, 52 Wk Low, Market Cap, EPS, P/E Ratio, and # Shares Out. However, I only want to deal with a few of the columns. Is there any way to only select a few of the columns to pull the data from? This is what I am using to pull the data and it works, but I don't want all of it:
try {
Class.forName("sun.jdbc.odbc.JdbcOdbcDriver");
Connection connect = DriverManager.getConnection("jdbc:odbc:Book1");
Statement st = connect.createStatement();
ResultSet result_set = st.executeQuery("SELECT * FROM A3:P30");
ResultSetMetaData md = result_set.getMetaData();
int columns = md.getColumnCount();
for (int i = 1; i <= columns; i++) {
columnNames.addElement(md.getColumnName(i));
}
while (result_set.next()) {
Vector row = new Vector(columns);
for (int i = 1; i <= columns; i++)
row.addElement(result_set.getObject(i));
data.addElement(row);
}
result_set.close();
st.close();
} catch (Exception e) {
System.out.println(e);
}
In Book1, I have 27 entries which is where P30 comes from..
Thank you, in advance!
If you have an header line like Last, Previous, Close, High etc use
select [Last],[Previous],[Close],[High] from [SheetName$]
We are using the same code and works fine.
In order to query the database meta data in Sybase ASE, I found this relevant answer (not the accepted one), to be ideal:
From a Sybase Database, how I can get table description ( field names and types)?
Unfortunately, I can't seem to find any documentation, how I'm supposed to call sp_help from JDBC. According to the documentation, sp_help returns several cursors / result sets. The first one contains information about the table itself, the second one about the columns, etc. When I do this:
PreparedStatement stmt = getConnection().prepareStatement("sp_help 't_language'");
ResultSet rs = stmt.executeQuery();
while (rs.next()) {
System.out.println(rs.getObject(1));
// ...
}
I only get the results from the first cursor. How to access the other ones?
When you have multiple result sets you need to use the execute() method rather than executeQuery().
Here's an example:
CallableStatement cstmt;
ResultSet rs;
int i;
String s;
...
cstmt.execute(); // Call the stored procedure 1
rs = cstmt.getResultSet(); // Get the first result set 2
while (rs.next()) { // Position the cursor 3
i = rs.getInt(1); // Retrieve current result set value
System.out.println("Value from first result set = " + i);
// Print the value
}
cstmt.getMoreResults(); // Point to the second result set 4a
// and close the first result set
rs = cstmt.getResultSet(); // Get the second result set 4b
while (rs.next()) { // Position the cursor 4c
s = rs.getString(1); // Retrieve current result set value
System.out.println("Value from second result set = " + s);
// Print the value
}
rs.close(); // Close the result set
cstmt.close(); // Close the statement
You also need to call getUpdateCount() as well as getMoreResults() to read the entire result set. Here is some code I used to call sp_helpartition to retrieve partition information from a SYBASE DB.
try {
connection = getPooledConnection(poolName);
statement = connection.createStatement();
CallableStatement callable = connection.prepareCall(
"{ call sp_helpartition(?) }");
callable.setString(1,tableName);
callable.execute();
int partitions = 0;
/*
* Loop through results until there are no more result sets or
* or update counts to read. The number of partitions is recorded
* in the number of rows in the second result set.
*/
for (int index = 0 ; ; index ++){
if (callable.getMoreResults()){
ResultSet results = callable.getResultSet();
int count = 0 ;
while (results.next()){
count++;
}
if (index == 1){
partitions = count;
}
} else if (callable.getUpdateCount() == -1){
break ;
}
}
return partitions ;
} catch (Exception e) {
return 0 ;
} finally {
statement.close();
connection.close();
}
Thanks to Martin Clayton's answer here, I could figure out how to query Sybase ASE's sp_help function generically. I documented some more details about how this can be done in my blog here. I worked support for multiple JDBC result sets into jOOQ. In the case of sp_help, calling that function using the jOOQ API might look like this:
Factory create = new ASEFactory(connection);
// Get a list of tables, a list of user types, etc
List<Result<Record>> tables = create.fetchMany("sp_help");
// Get some information about the my_table table, its
// columns, keys, indexes, etc
List<Result<Record>> results = create.fetchMany("sp_help 'my_table'");
Shouldn't this be a pretty straightforward operation? However, I see there's neither a size() nor length() method.
Do a SELECT COUNT(*) FROM ... query instead.
OR
int size =0;
if (rs != null)
{
rs.last(); // moves cursor to the last row
size = rs.getRow(); // get row id
}
In either of the case, you won't have to loop over the entire data.
ResultSet rs = ps.executeQuery();
int rowcount = 0;
if (rs.last()) {
rowcount = rs.getRow();
rs.beforeFirst(); // not rs.first() because the rs.next() below will move on, missing the first element
}
while (rs.next()) {
// do your standard per row stuff
}
Well, if you have a ResultSet of type ResultSet.TYPE_FORWARD_ONLY you want to keep it that way (and not to switch to a ResultSet.TYPE_SCROLL_INSENSITIVE or ResultSet.TYPE_SCROLL_INSENSITIVE in order to be able to use .last()).
I suggest a very nice and efficient hack, where you add a first bogus/phony row at the top containing the number of rows.
Example
Let's say your query is the following
select MYBOOL,MYINT,MYCHAR,MYSMALLINT,MYVARCHAR
from MYTABLE
where ...blahblah...
and your output looks like
true 65537 "Hey" -32768 "The quick brown fox"
false 123456 "Sup" 300 "The lazy dog"
false -123123 "Yo" 0 "Go ahead and jump"
false 3 "EVH" 456 "Might as well jump"
...
[1000 total rows]
Simply refactor your code to something like this:
Statement s=myConnection.createStatement(ResultSet.TYPE_FORWARD_ONLY,
ResultSet.CONCUR_READ_ONLY);
String from_where="FROM myTable WHERE ...blahblah... ";
//h4x
ResultSet rs=s.executeQuery("select count(*)as RECORDCOUNT,"
+ "cast(null as boolean)as MYBOOL,"
+ "cast(null as int)as MYINT,"
+ "cast(null as char(1))as MYCHAR,"
+ "cast(null as smallint)as MYSMALLINT,"
+ "cast(null as varchar(1))as MYVARCHAR "
+from_where
+"UNION ALL "//the "ALL" part prevents internal re-sorting to prevent duplicates (and we do not want that)
+"select cast(null as int)as RECORDCOUNT,"
+ "MYBOOL,MYINT,MYCHAR,MYSMALLINT,MYVARCHAR "
+from_where);
Your query output will now be something like
1000 null null null null null
null true 65537 "Hey" -32768 "The quick brown fox"
null false 123456 "Sup" 300 "The lazy dog"
null false -123123 "Yo" 0 "Go ahead and jump"
null false 3 "EVH" 456 "Might as well jump"
...
[1001 total rows]
So you just have to
if(rs.next())
System.out.println("Recordcount: "+rs.getInt("RECORDCOUNT"));//hack: first record contains the record count
while(rs.next())
//do your stuff
int i = 0;
while(rs.next()) {
i++;
}
I got an exception when using rs.last()
if(rs.last()){
rowCount = rs.getRow();
rs.beforeFirst();
}
:
java.sql.SQLException: Invalid operation for forward only resultset
it's due to by default it is ResultSet.TYPE_FORWARD_ONLY, which means you can only use rs.next()
the solution is:
stmt=conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
[Speed consideration]
Lot of ppl here suggests ResultSet.last() but for that you would need to open connection as a ResultSet.TYPE_SCROLL_INSENSITIVE which for Derby embedded database is up to 10 times SLOWER than ResultSet.TYPE_FORWARD_ONLY.
According to my micro-tests for embedded Derby and H2 databases it is significantly faster to call SELECT COUNT(*) before your SELECT.
Here is in more detail my code and my benchmarks
The way of getting size of ResultSet, No need of using ArrayList etc
int size =0;
if (rs != null)
{
rs.beforeFirst();
rs.last();
size = rs.getRow();
}
Now You will get size, And if you want print the ResultSet, before printing use following line of code too,
rs.beforeFirst();
It is a simple way to do rows-count.
ResultSet rs = job.getSearchedResult(stmt);
int rsCount = 0;
//but notice that you'll only get correct ResultSet size after end of the while loop
while(rs.next())
{
//do your other per row stuff
rsCount = rsCount + 1;
}//end while
String sql = "select count(*) from message";
ps = cn.prepareStatement(sql);
rs = ps.executeQuery();
int rowCount = 0;
while(rs.next()) {
rowCount = Integer.parseInt(rs.getString("count(*)"));
System.out.println(Integer.parseInt(rs.getString("count(*)")));
}
System.out.println("Count : " + rowCount);
theStatement=theConnection.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_READ_ONLY);
ResultSet theResult=theStatement.executeQuery(query);
//Get the size of the data returned
theResult.last();
int size = theResult.getRow() * theResult.getMetaData().getColumnCount();
theResult.beforeFirst();
I checked the runtime value of the ResultSet interface and found out it was pretty much a ResultSetImpl all the time. ResultSetImpl has a method called getUpdateCount() which returns the value you are looking for.
This code sample should suffice:
ResultSet resultSet = executeQuery(sqlQuery);
double rowCount = ((ResultSetImpl)resultSet).getUpdateCount()
I realize that downcasting is generally an unsafe procedure but this method hasn't yet failed me.
Today, I used this logic why I don't know getting the count of RS.
int chkSize = 0;
if (rs.next()) {
do { ..... blah blah
enter code here for each rs.
chkSize++;
} while (rs.next());
} else {
enter code here for rs size = 0
}
// good luck to u.
I was having the same problem. Using ResultSet.first() in this way just after the execution solved it:
if(rs.first()){
// Do your job
} else {
// No rows take some actions
}
Documentation (link):
boolean first()
throws SQLException
Moves the cursor to the first row in this ResultSet object.
Returns:
true if the cursor is on a valid
row; false if there are no rows in the result set
Throws:
SQLException - if a database access error occurs; this method is called on a closed result set or the result set type is TYPE_FORWARD_ONLY
SQLFeatureNotSupportedException - if the JDBC driver does not support
this method
Since:
1.2
Easiest approach, Run Count(*) query, do resultSet.next() to point to the first row and then just do resultSet.getString(1) to get the count. Code :
ResultSet rs = statement.executeQuery("Select Count(*) from your_db");
if(rs.next()) {
int count = rs.getString(1).toInt()
}
Give column a name..
String query = "SELECT COUNT(*) as count FROM
Reference that column from the ResultSet object into an int and do your logic from there..
PreparedStatement statement = connection.prepareStatement(query);
statement.setString(1, item.getProductId());
ResultSet resultSet = statement.executeQuery();
while (resultSet.next()) {
int count = resultSet.getInt("count");
if (count >= 1) {
System.out.println("Product ID already exists.");
} else {
System.out.println("New Product ID.");
}
}