Is there an efficient way to obtain a list (preferably an array, a ResultSet will do) to SELECT a lot of rows.
For example:
Connection con = DriverManager.getConnection(host,username,password);
String sql = "SELECT * FROM table_name WHERE food = ? AND expiration > ?";
PreparedStatement stmt = con.prepareStatement(sql);
Using the above code, I want to get all the food from a given array that isn't expired.
String[] foodList = {"banana","apple","orange",...}
where the expiration date is a constant date (lets say 3 days ago). However, the way I have it is that the String and PreparedStatement are in a for loop that loop the number of foods in the array to individually check the expiration date. This creates a lot of ResultSets after I execute each individually.
Most SQL Databases support a IN (list) expression. This is roughly equivalent to providing a or expression:
SELECT id FROM table WHERE food IN ('Apple', 'Banana') AND exp < ?
is similar to
SELECT id FROM table WHERE (food = 'Apple' or food = 'Banana') AND exp < ?
In both cases some RDBMS can optimize it.
However first of all there is a limitation in the number of list items you can specify in the IN or number of characters you can use in the statement. So if your list can be variable long you need to be prepared to run multiple statements.
Secondly you cannot* set a array as an argument to a PreparedStatement and expect it to work with IN.
Unfortunately in plain JDBC all you can do is to concatenate a String. This is frowned upon, but there is no good alternative (unless you want to do something like giving the list of foods as a single list and use a "instring" expression).
Make sure to add as many ? (but not too many) as you expect parameters and then set them in the IN:
String[] foods = ...;
int remain = foods.length;
int start = 0;
while(remain > 0)
{ if (remain >= 100)
executeBatch(foods, start, 100); start+=100; remain-=100;
else if (remain >= 30)
executeBatch(foods, start, 30); start+=30; remain-=30;
else {
executeBatch(foods, start, 1); start+=1; remain-=1;
}
}
void executeBatch(String[] f, int off, int len)
{
StringBuilder sqlBuf = StringBuilder("... IN(");
for(int i=0;i<len;i++) {
sqlBuf.append((i!=0)?",?":"?");
}
String sql = sqlBuf.append(") AND exp < ?").toString();
PreparedStatement ps = c.prepareStatement(sql);
for(int i=0;i<foods.length;i++)
ps.setString(i+1, foods[i+off]);
ps.setTimestamp(foods.length+1, now);
....
}
This avoids to generate a lot of different SQL statement to compile. (Only 100,30 or 1 ?)). You can use the same logic for the OR case.
* not to be confused with ARRAY database type.
Probably not the most elegant solution, and you won't get any performance benefit from the prepared statement (but you will get parameter binding):
StringBuilder sql = new StringBuilder("SELECT * FROM table_name WHERE expiration > ? AND food IN (");
for (int i = 0; i < foodList.length; i++) {
if (i > 0) {
sql.append(',');
}
sql.append('?');
}
sql.append(")");
Connection con = DriverManager.getConnection(host, username, password);
PreparedStatement stmt = con.prepareStatement(sql.toString());
stmt.setDate(1, expirationDate);
for (int i = 0; i < foodList.length; i++) {
stmt.setString(i + 2, foodList[i]);
}
ResultSet rs = stmt.executeQuery();
/* ... Do Stuff ... */
Related
I want to insert list of string into database so usually we will store the string directly into the database by either using a prepared statement or a batch statement. Now I want to insert list of string into a database, so I have used a prepared statement.
List<String> Account_Number = Files.lines(Paths.get("D:\\PDFTOEXCEL\\Extractionfrompdf.txt"))
.filter(s -> s.contains(arra.get(7)))
.map(s -> s.split(":")[1].trim())
.collect(Collectors.toList());
System.out.println(Account_Number);
try {
Connection conn = PDFTOEXCEL.getConnection();
PreparedStatement stmt = conn.prepareStatement("insert into client_info values(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)");
stmt.setString(1, Account_Number);
int k = stmt.executeUpdate();
I have about 31 columns in my database. Just for showing I have posted only one in this code. All are in the list of the string only.
Have you tried to set the parameters one by one? Something like this:
int size = Account_Number.size();
for (int i = 1; i <= size; i++) {
stmt.setString(i, Account_Number.get(i-1));
}
Am I missing something obvious here?
I have the following method to execute queries against an SQLite local database using org.xerial:sqlite-jdbc:3.21.0.
public List<HashMap<String, Object>> executeQuery(String sql, List<Object> vals) throws Exception {
List<HashMap<String, Object>> rows = new ArrayList<>();
Connection conn = getConnection();
try (PreparedStatement stmnt = conn.prepareStatement(sql)) {
if (!vals.isEmpty()) {
for (int i = 0; i < vals.size(); i++) {
stmnt.setObject(i + 1, vals.get(i));
}
}
ResultSet rs = stmnt.executeQuery();
ResultSetMetaData meta = rs.getMetaData();
HashMap<String, Object> row;
while (rs.next()) {
row = new HashMap<>();
for (int i = 0; i < meta.getColumnCount(); i++) {
row.put(meta.getColumnName(i + 1), rs.getObject(i + 1));
}
rows.add(row);
}
} finally {
putConnection(conn);
}
return rows;
}
However, when I pass the following SQL into the method along with the following values, they don't get set (but it also doesn't throw an exception). It's like it internally assigns it but forgets to tell the database.
SELECT * FROM 'airlines' WHERE 'name' LIKE ? LIMIT 1
vals: size = 1 {"MyAirline"}
I can see from debugging that it gets inside the loop to setObject.
In ANSI standard SQL, single quotes (') are used to delimit literal strings and double quotes (") are used to delimit table/column names. So
SELECT * FROM 'airlines' WHERE 'name' LIKE ? LIMIT 1
really means "select all columns from the literal string 'airlines' where the literal string 'name' matches the pattern supplied by the parameter".
Interestingly, SQLite seems to be clever enough to interpret the literal string 'airlines' as the table name "airlines" but it is still interpreting 'name' as a literal string. Therefore, for every row in "airlines" it is comparing the literal string 'name' to the string value 'MyAirline' and it never matches, so the ResultSet contains no rows.
Your SQL command text should be
SELECT * FROM "airlines" WHERE "name" LIKE ? LIMIT 1
so SQLite will compare the contents of the "name" column with the value 'MyAirline'.
When i run the below program
package com.util;
import java.util.ArrayList;
public class Test {
public static void main(String[] args) throws Exception {
ArrayList<String> list_of_symbols = new ArrayList<String>();
list_of_symbols.add("ABB");
list_of_symbols.add("ACC");
list_of_symbols.add("SBIN");
StringBuilder sb_builder = new StringBuilder();
for (int i = 0; i < list_of_symbols.size(); i++) {
sb_builder.append(list_of_symbols.get(i) + ",");
}
String sql = "Select * from data where symbol_name IN ("
+ sb_builder.deleteCharAt(sb_builder.length() - 1).toString()
+ ")";
System.out.println(sql);
}
}
The Result of SQL IS
Select * from data where symbol_name IN (ABB,ACC,SBIN)
Where as the expected result should be
Select * from data where symbol_name IN ('ABB','ACC','SBIN')
Could you please let me know how can i keep Quotes so that it becomes valid SQL
Don't use string concatenation to fill in SQL parameters. It's error-prone. Instead, build the SQL with as many ? as you need, and then use a PreparedStatement and as many setString(x, theString) as you need to fill in the ?.
In your case, it would look roughly like this:
package com.util;
import java.util.ArrayList;
public class Test {
public static void main(String[] args) throws Exception {
ArrayList<String> list_of_symbols = new ArrayList<String>();
list_of_symbols.add("ABB");
list_of_symbols.add("ACC");
list_of_symbols.add("SBIN");
// Build the statement
StringBuilder sql = new StringBuilder(200);
sql.append("Select * from data where symbol_name IN (");
for (int i = 0; i < list_of_symbols.size(); i++) {
sql.append(i == 0 ? "?" : ", ?");
}
sql.append(')');
// Build the PreparedStatement and fill in the parameters
PreparedStatement ps = someConnection.prepareStatement(sql.toString());
for (int i = 0; i < list_of_symbols.size(); i++) {
ps.setString(i + 1, list_of_symbols.get(i));
}
// Do it
ResultSet rs = ps.executeQuery();
}
}
(This is un-optimized, and dashed off. Some editing is likely required.)
This site has a good explanation of why using string concat for parameters is a bad idea, as well as practical examples of how to do things properly in many languages, including Java.
As you know the number of values in your list I'd suggest to put in one ? per IN value and make it a PreparedStatement.
Then simply do
ps.setString (n, nthString);
In a loop for 1..n parameters and your driver will handle proper escaping.
Like this:
List<String> list_of_symbols = new ArrayList<String>();
list_of_symbols.add("ABB");
list_of_symbols.add("ACC");
list_of_symbols.add("SBIN");
StringJoiner join = new StringJoiner(",",
"Select * from data where symbol_name IN (", ")");
for (int i = 0; i < list_of_symbols.size(); i++) {
join.add("?");
}
PreparedStatement ps = aConnection.prepareStatement(join.toString());
for (int i = 0; i < list_of_symbols.size(); i++) {
ps.setString(i+1, list_of_symbols.get(i));
}
When making your list, use this:
sb_builder.append("'"+list_of_symbols.get(i) + "',");
Notice the ' on the string. This works for simple cases only, where strings do not have ' and you're not worried with SQL Injection. For more complex cases, used PreparedStatements and add a list of ?. Then replace the ? with the Strings you want to use.
My objective is this:
I have a SELECT query with that I want to run on a database with a lot of rows. This query will produce a LOT of results, so I was thinking of running it in iterations with LIMIT 1000 and OFFSET ? where ? will be the last row that were processed in each iteration.
Eg:
Get rows 1 - 1000
Get rows 1001 - 2000
Get rows 2001 - 3000
Get rows 3001 - 4000
...
I was thinking about doing it in a loop where the each new iteration will set the last iteration's row as the new iteration's OFFSET (Eg: OFFSET 1001, OFFSET 2001, etc. as shown above).
I am new to using JDBC, so is this the correct way to do it? If so, how do I re-use PreparedStatement when I must execute it and get the result for each iteration?
If this isn't the correct way to do it, what is the correct way?
EDIT:
Here's my current code:
private static void import(Date from, Date to) throws Exception {
PreparedStatement p = connect.prepareStatement(statement);
p.setInt(1, 0);
p.add
ResultSet results;
for (int i=0; i< WAVES; i++) {
results = p.executeQuery();
Integer lastRow = importFrom(results);
p.setInt(1, lastRow.intValue()+1);
results.close();
}
p.close();
}
EDIT 2:
Here's the SQL String:
SELECT
item.aitem_id, item.action_type, item.user_id, item.pid, item.pr_id, item.action_time, item.notes, item.screen, item.vid, item.lo_id,
vals.value_name, vals.simple_int_value, vals.simple_double_value, vals.simple_date_value, vals.simple_string_value,
data.version_id, data.prev_version_id
FROM mySchema.aitems item
JOIN mySchema.avalues vals ON item.aitem_id=vals.aitem_id
JOIN mySchema.adata data ON item.aitem_id=data.aitem_id
LIMIT 1000 OFFSET ?;
I modified both my code and SQL to prepare with only the OFFSET.
What your doing is correct but It would be a good idea to add a call to clearParameters() and uses try finally blocks. Below is how I would implement it
PreparedStatement p = null;
ResultSet results = null;
Integer lastRow = 0;
try
{
p = connect.prepareStatement(statement);
p.add
for (int i=0; i< WAVES; i++)
{
p.clearParameters();
p.setInt(1,lastRow.intValue() + 1)
try // This try might not really be necessary
{
results = p.executeQuery();
Integer lastRow = importFrom(results);
p.setInt(1, lastRow.intValue()+1);
} // Add dealing with exceptions
finally
{
results.close();
}
}
} //Add dealing with exceptions
finally
{
p.close();
}
Is there a way to pass an array of strings for a "WHERE country IN (...)" query?
something like this:
String[] countries = {"France", "Switzerland"};
PreparedStatement pstmt = con.prepareStatement("SELECT * FROM table WHERE country IN (?...)");
pstmt.setStringArray(1, countries);
pstmt.executeQuery();
an ugly workaround would be to create the query based on the size of the array
String[] countries = {"France", "Switzerland"};
if (countries.size() == 0) { return null; }
String query = "SELECT * FROM table WHERE country IN (?";
for (int i = 1; i < countries.size; i++) { query += ", ?"; }
PreparedStatement pstmt = con.prepareStatement(query);
for (int i = 0; i < countries.size; i++) { pstmt.setString(1+i, countries[i]); }
pstmt.executeQuery();
but this looks really ugly.
any idea?
No, it's not possible. ORMs like Hibernate or wrapper APIs like Spring JDBC allows doing that. But with plain JDBC, you must do it yourself.
I think the work around would be formulating the entire query string at runtime and using a Statement object instead of PreparedStatement.
No way to try this.See here to other ways.But there is one exception, if you use oracle database then you can try this
If your database engine support IN (subquery), you can create a view or memory table to do it.