Develop Reference

Insert list <string> into database using Java

I want to insert list of string into database so usually we will store the string directly into the database by either using a prepared statement or a batch statement. Now I want to insert list of string into a database, so I have used a prepared statement.
List<String> Account_Number = Files.lines(Paths.get("D:\\PDFTOEXCEL\\Extractionfrompdf.txt"))
.filter(s -> s.contains(arra.get(7)))
.map(s -> s.split(":")[1].trim())
.collect(Collectors.toList());
System.out.println(Account_Number);
try {
Connection conn = PDFTOEXCEL.getConnection();
PreparedStatement stmt = conn.prepareStatement("insert into client_info values(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)");
stmt.setString(1, Account_Number);
int k = stmt.executeUpdate();
I have about 31 columns in my database. Just for showing I have posted only one in this code. All are in the list of the string only.

Have you tried to set the parameters one by one? Something like this:
int size = Account_Number.size();
for (int i = 1; i <= size; i++) {
stmt.setString(i, Account_Number.get(i-1));
}

Insert performance tuning

Currently we are selecting data from one database and inserting it into a backup database(SQL SERVER).
This data always contains more than 15K records in one select.
We are using Enumeration to iterate over the data selected.
We are using JDBC PreparedStatement to insert data as:
Enumeration values = ht.elements(); -- ht is HashTable containing selected data.
while(values.hasMoreElements())
{
pstmt = conn.prepareStatement("insert query");
pstmt.executeUpdate();
}
I am not sure if this is the correct or efficient way to do the faster insert.
For inserting 10k rows it takes near about 30 min or more.
Is there any efficient way to make it fast?
Note: Not using any indexes on the table.

Use a batch insert, but commit after a few entris, don't try to send all 10K at once. Try investigating to get the best size, it' a trade off to memory vs network trips.
Connection connection = new getConnection();
Statement statement = connection.createStatement();
int i = 0;
for (String query : queries) {
statement.addBatch("insert query");
if ((i++ % 500) == 0) {
// Do an execute now and again, don't send too many at once
statement.executeBatch();
}
}
statement.executeBatch();
statement.close();
connection.close();
Also, from your code I'm not sure what you are doing, but use paramaterised queries rather than sending 10K insert statements as text. Something like:
String q= "INSERT INTO data_table (id) values (?)";
Connection connection = new getConnection();
PreparedStatement ps = connection.prepareStatement(q);
for (Data d: data) {
ps.setString(1, d.getId());
ps.addBatch();
}
ps.executeBatch();
ps.close();
connection.close();

You can insert all the values in one sql command:
INSERT INTO Table1 ( Column1, Column2 ) VALUES
( V1, V2 ), ( V3, V4 ), .......
You may also insert the values by bulks of 500 records, for example, if the query would become very big. It is not efficient at all to insert on row per statement remotely (using a connection). Another solution is to do the inserts using a stored procedure. You just pass the values to it as parameters.
Here is how you can do it using the INSERT command above:
Enumeration values = ht.elements(); -- ht is HashTable containing selected data.
int i=0;
String sql="";
while(values.hasMoreElements())
{
sql+="(" + values + ")"; //better use StringBuffer here
i++;
if(i % 500 == 0) {
pstmt = conn.prepareStatement("insert query "+sql);
pstmt.executeUpdate();
sql="";
}
else
sql += " , ";
}

Efficient way to select data with a single condition

Is there an efficient way to obtain a list (preferably an array, a ResultSet will do) to SELECT a lot of rows.
For example:
Connection con = DriverManager.getConnection(host,username,password);
String sql = "SELECT * FROM table_name WHERE food = ? AND expiration > ?";
PreparedStatement stmt = con.prepareStatement(sql);
Using the above code, I want to get all the food from a given array that isn't expired.
String[] foodList = {"banana","apple","orange",...}
where the expiration date is a constant date (lets say 3 days ago). However, the way I have it is that the String and PreparedStatement are in a for loop that loop the number of foods in the array to individually check the expiration date. This creates a lot of ResultSets after I execute each individually.

Most SQL Databases support a IN (list) expression. This is roughly equivalent to providing a or expression:
SELECT id FROM table WHERE food IN ('Apple', 'Banana') AND exp < ?
is similar to
SELECT id FROM table WHERE (food = 'Apple' or food = 'Banana') AND exp < ?
In both cases some RDBMS can optimize it.
However first of all there is a limitation in the number of list items you can specify in the IN or number of characters you can use in the statement. So if your list can be variable long you need to be prepared to run multiple statements.
Secondly you cannot* set a array as an argument to a PreparedStatement and expect it to work with IN.
Unfortunately in plain JDBC all you can do is to concatenate a String. This is frowned upon, but there is no good alternative (unless you want to do something like giving the list of foods as a single list and use a "instring" expression).
Make sure to add as many ? (but not too many) as you expect parameters and then set them in the IN:
String[] foods = ...;
int remain = foods.length;
int start = 0;
while(remain > 0)
{ if (remain >= 100)
executeBatch(foods, start, 100); start+=100; remain-=100;
else if (remain >= 30)
executeBatch(foods, start, 30); start+=30; remain-=30;
else {
executeBatch(foods, start, 1); start+=1; remain-=1;
}
}
void executeBatch(String[] f, int off, int len)
{
StringBuilder sqlBuf = StringBuilder("... IN(");
for(int i=0;i<len;i++) {
sqlBuf.append((i!=0)?",?":"?");
}
String sql = sqlBuf.append(") AND exp < ?").toString();
PreparedStatement ps = c.prepareStatement(sql);
for(int i=0;i<foods.length;i++)
ps.setString(i+1, foods[i+off]);
ps.setTimestamp(foods.length+1, now);
....
}
This avoids to generate a lot of different SQL statement to compile. (Only 100,30 or 1 ?)). You can use the same logic for the OR case.
* not to be confused with ARRAY database type.

Probably not the most elegant solution, and you won't get any performance benefit from the prepared statement (but you will get parameter binding):
StringBuilder sql = new StringBuilder("SELECT * FROM table_name WHERE expiration > ? AND food IN (");
for (int i = 0; i < foodList.length; i++) {
if (i > 0) {
sql.append(',');
}
sql.append('?');
}
sql.append(")");
Connection con = DriverManager.getConnection(host, username, password);
PreparedStatement stmt = con.prepareStatement(sql.toString());
stmt.setDate(1, expirationDate);
for (int i = 0; i < foodList.length; i++) {
stmt.setString(i + 2, foodList[i]);
}
ResultSet rs = stmt.executeQuery();
/* ... Do Stuff ... */

java: retrieve keys after executeBatch() in H2

I am trying to retrieve generated keys from an executeBatch() transaction but I only get the last key to be added.
this is my code:
PreparedStatement ps_insert = conn.prepareStatement(insertQuery, PreparedStatement.RETURN_GENERATED_KEYS);
for (int i = 0 ; i < adding_dates.length ; i++){
ps_insert.setInt(1, Integer.parseInt(consultant_id));
ps_insert.setDate(2, adding_dates[i]);
ps_insert.setInt(3, Integer.parseInt(room_id));
ps_insert.addBatch();
}
ps_insert.executeBatch();
ResultSet rs = ps_insert.getGeneratedKeys(); //<-- Only the last key retrieved
conn.commit();
What am I doing wrong?
EDIT: Apologies for not mentioning that I use H2 (http://www.h2database.com/html/main.html) database in embedded mode.

According to H2 jdbc driver javadocs, this is the normal behaviour:
Return a result set that contains the last generated auto-increment
key for this connection, if there was one. If no key was generated by
the last modification statement, then an empty result set is returned.
The returned result set only contains the data for the very last row.

You must iterate the ResultSet to retrieve the keys.
PreparedStatement ps_insert = conn.prepareStatement(insertQuery, PreparedStatement.RETURN_GENERATED_KEYS);
for (int i = 0 ; i < adding_dates.length ; i++){
ps_insert.setInt(1, Integer.parseInt(consultant_id));
ps_insert.setDate(2, adding_dates[i]);
ps_insert.setInt(3, Integer.parseInt(room_id));
ps_insert.addBatch();
}
ps_insert.executeBatch();
ResultSet rs = ps_insert.getGeneratedKeys(); //<-- Only the last key retrieved
if (rs.next()) {
ResultSetMetaData rsmd = rs.getMetaData();
int colCount = rsmd.getColumnCount();
do {
for (int i = 1; i <= colCount; i++) {
String key = rs.getString(i);
System.out.println("key " + i + "is " + key);
}
}
while (rs.next();)
}
conn.commit();

This is a limitation of H2 implementation. This is an issue.
For now use inserts/updates without batch, or query generated keys somehow through select.

If you are sharing a session/connection between 2 threads, and two of those threads try to execute statements at the same time, then you might see this kind of problem.
You probably need to either (a) use a connection pool or (b) synchronise your entire access to the DB.
for instance for option (b)
put a synchronize token infront of your method to make it thread safe
Just a thought as i dont know you complete execution context

JDBC batch query for high performance

I want to do batch query DB for high performance, example sql to query based on different customer_id:
select order_id,
cost
from customer c
join order o using(id)
where c.id = ...
order by
I'm not sure how to do it using JDBC statement. I know I can use stored procedure for this purpose, but it's much better if I can just write sql in Java app instead of SP.
I'm using DBCP for my Java client and MySQL DB.

The JDBC Specification 4.0 describes a mechanism for batch updates. As such, the batch features in JDBC can be used for insert or update purposes. This is described in chapter 14 of the specification.
AFAIK there is not a mechanism for select batches, probably because there is no apparent need for that since, as others have recommended, you can simply retrieve all the rows that you want at once by properly constructing your query.
int[] ids = { 1, 2, 3, 4 };
StringBuilder sql = new StringBuilder();
sql.append("select jedi_name from jedi where id in(");
for (int i = 0; i < ids.length; i++) {
sql.append("?");
if(i+1 < ids.length){
sql.append(",");
}
}
sql.append(")");
System.out.println(sql.toString());
try (Connection con = DriverManager.getConnection(...)) {
PreparedStatement stm = con.prepareStatement(sql.toString());
for(int i=0; i < ids.length; i++){
stm.setInt(i+1, ids[i]);
}
ResultSet rs = stm.executeQuery();
while (rs.next()) {
System.out.println(rs.getString("jedi_name"));
}
} catch (SQLException e) {
e.printStackTrace();
}
Output
select jedi_name from jedi where id in(?,?,?,?)
Luke, Obiwan, Yoda, Mace Windu
Is there any reason why you would consider that you need a thing like a batch-select statement?

It is really does not matter what is your SQL statement (you can use as many nested joins as your DB can handle). Below is basic Java example (not DBCP). For DBCP example which is pretty similar you can check out their example.
Connection connect = DriverManager.getConnection(YOUR_CONNECTION_STRING);
// Statements allow to issue SQL queries to the database
Statement statement = connect.createStatement();
ResultSet resultSet = statement.executeQuery("select order_id, cost
from customer c
join order o using(id)
where c.id = ...
order by");