Update in bulk through JDBC batch gives SQLException TransactionImpl - java

I'm using EJB3 with Oracle database and JDBC.
I'm working on an app where I have to fire 25000 UPDATE queries.
My code is as follows:
public int updateStatus(List<String> idList) {
Connection connection = getConnection(); // Connection initialized properly for oracle db
statement = connection.createStatement();
String sql = null;
for (String id : idlist) { // idList is properly filled
sql = "UPDATE TBLTEST SET STATUS = 'FIXED' WHERE ID = '" + id + "'";
statement.addBatch(sql);
}
int[] affectedRecords = statement.executeBatch();
}
Please note, the class in which this method is written, is annotated as
#TransactionManagement(TransactionManagementType.CONTAINER)
This code is working perfectly fine upto 8000 queries. For more ids, it throws the following exception:
org.jboss.util.NestedSQLException: Transaction TransactionImple < ac, BasicAction: 0:ffffc0a80272:1652:56bd6be5:57e status: ActionStatus.ABORTED > cannot proceed STATUS_ROLLEDBACK; - nested throwable: (javax.transaction.RollbackException: Transaction TransactionImple < ac, BasicAction: 0:ffffc0a80272:1652:56bd6be5:57e status: ActionStatus.ABORTED > cannot proceed STATUS_ROLLEDBACK)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.checkTransactionActive(WrapperDataSource.java:165)
at org.jboss.resource.adapter.jdbc.WrappedConnection.checkTransactionActive(WrappedConnection.java:843)
at org.jboss.resource.adapter.jdbc.WrappedConnection.checkStatus(WrappedConnection.java:858)
at org.jboss.resource.adapter.jdbc.WrappedConnection.checkTransaction(WrappedConnection.java:835)
at org.jboss.resource.adapter.jdbc.WrappedConnection.createStatement(WrappedConnection.java:183)
Can anyone help with the exception?

Best guess: By using individual SQL statements instead of PreparedStatement you force the driver to send all your statements (> 400k of character data) to the DB and the DB to parse all of that 400k characters which will hit a limit at some time and breaks things (Exception is not clear on where or what broke as it hides the causing Exception).
How to fix:
Go for individual batches of "not too many" statements at a time - say... 1000:
public int updateStatus(List<String> idList) {
List<Integer> affectedRecords = new ArrayList<Integer>(idList.size());
try(Connection connection = getConnection();
Statement statement = connection.createStatement()) {
int count = 0;
for (String id : idList) {
statement.addBatch("UPDATE TBLTEST SET STATUS = 'FIXED' WHERE ID = '" + id + "'");
//Execute after 1000 rows
if(++count % 1000 == 0) {
int[] result = statement.executeBatch();
//Utility Method - you need to implement to add int[] into the List
addResults(affectedRecords, result);
statement.clearBatch();
}
}
//In need of final execute?
if(count % 1000 > 0) {
// For good measure execute once more
int[] result = statement.executeBatch();
//Utility Method - you need to implement to add int[] into the List
addResults(affectedRecords, result);
}
} catch(SQLException se) {
se.printStackTrace();
}
}

Related

JdbcTemplate.batchUpdate() returns 0, on insert error for one item but inserts the remaining item into sql server db despite using #Transactional

My code is very similar to one below, despite configuring the transaction manager, except for the incorrect item all items are inserted into the db. This is absurd as either there should be all insert or none using #Transactional.
List<Book> books = new ArrayList();
for (int count = 0; count < size; count++) {
if (count == 500) {
// Create an invalid data for id 500, test rollback
// Name max 255, this book has length of 300
books.add(new Book(NameGenerator.randomName(300), new BigDecimal(1.99)));
continue;
}
books.add(new Book(NameGenerator.randomName(20), new BigDecimal(1.99)));
}
#Transactional
public int[][] batchInsert(List<Book> books, int batchSize) {
int[][] updateCounts = jdbcTemplate.batchUpdate(
"insert into books (name, price) values(?,?)",
books,
batchSize,
new ParameterizedPreparedStatementSetter<Book>() {
public void setValues(PreparedStatement ps, Book argument) throws SQLException {
ps.setString(1, argument.getName());
ps.setBigDecimal(2, argument.getPrice());
}
});
return updateCounts;
}
This is a difficult situation I facing for batch update. The actual solution lies in using prepared statement and not JdbcTemplate(Actually under the hood jdbcTemplate uses prepared statement). The scenario was to insert 5000 records in db in distrubuted application with one batch(JSON Payload) of 1000 records.
On the connection object one needs to turn off auto commit and then commit the transaction if all the insert happens successfully. The scenario is tried and tested in Spring boot.
Connection conn = dataSource.getConnection(); //Configure
datasource in the config class
conn.setAutoCommit(false); //The most important part of the code
PreparedStatement pstmt = conn.prepareStatement(
"INSERT INTO customers (CustID, Last_Name, " +
"First_Name, Email, Phone_Number)" +
" VALUES(?,?,?,?,?)");
for (int i = 0; i < firstNames.length; i++) {
// Add each parameter to the row.
pstmt.setInt(1, i + 1);
pstmt.setString(2, lastNames[i]);
pstmt.setString(3, firstNames[i]);
pstmt.setString(4, emails[i]);
pstmt.setString(5, phoneNumbers[i]);
// Add row to the batch.
pstmt.addBatch();
}
try {
// Batch is ready, execute it to insert the data
pstmt.executeBatch();
conn.commit(); //Any exception in inserting a record the commit is
skipped
} catch (SQLException e) {
System.out.println("Error message: " + e.getMessage());
return; // Exit if there was an error
}
Using prepared statement actually gives the control over the transaction and has far better performance than using JdbcTemaplte.
Also explained in the article https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/ConnectingToVertica/ClientJDBC/BatchInsertsUsingJDBCPreparedStatements.htm

Why does my for loop in android studio stop after the first iteration?

I created a system in which I can run all my postgre sql queries with only one single Async Task Class in Android Studio. This was really(!!) challenging due to the big amount of limitations that I had to face. But this works actually really great!
//Used for connecting to database and executing queries.
//Index 0 of input string must be the query, Index 1 must be the tablename we demand
//We can only gather data from 1 table for each query, so if you need data from several tablecolumns, use multiple queries like:
//[0] = query, [1] = tablename, [2] = 2nd query, [3] = 2nd tablename, [4] = 3rd query, [5] = 3rd table name ... and so on (each query must come with a tablename)
public class DBHandler extends AsyncTask<String, Void, List<String>>
{
public AsyncResponse delegate;
#Override
protected List<String> doInBackground(String...query)
{
List<String> result = new ArrayList<String>();
String sql;
String tableresult = null;
Connection conn = null;
Statement st = null;
ResultSet rs = null;
try {
Class.forName("org.postgresql.Driver");
conn = DriverManager.getConnection("jdbc:postgresql://192.168.200.300:5439/dbname?user=anonymous&password=secretpw");
st = conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_UPDATABLE); //necessary if you want to use rs.first() after rs.next(), it makes the resultset scrollable
for (int i = 0; i <= query.length-1; i = i+2) //queries are always stored in i=0 and/or in i+2, because i+1 contain the demanded tablenames for resultset handling
{
System.out.println("I is: " +i);
if (!query[i].isEmpty())
{
System.out.println(query[i]);
sql = query[i];
rs = st.executeQuery(sql);
while (rs.next())
if (!query[i + 1].isEmpty() || !rs.getString(query[i + 1]).isEmpty()) //if i+1 is empty, there is no demanded tablename. Used when we dont need any return values (ie. INSERT, UPDATE)
result.add(rs.getString(query[i + 1])); //demanded tablename is always stored in i+1
//We add an empty entry if we demand multiple tablenames so we can keep them seperate
//Might be replaced with any other char, but you will have to backtrack all usages of DBHandler and fix the filters there
if(i+2 < query.length)
result.add(" ");
}
rs.first(); //reset pointer for rs.next()
}
rs.close();
st.close();
conn.close();
System.out.println("End of AsyncTask");
}
catch (SQLException ex)
{
ex.printStackTrace();
}
catch (Exception e)
{
e.printStackTrace();
}
return result;
}
//onPostExecute returns query result in a List.
//We need to use interaces delegate feature to send the result to other classes, like "Auslieferung", which is implementing the interface
#Override
protected void onPostExecute(List<String> result)
{
super.onPostExecute(result);
System.out.println("Result: " +result.toString());
if (!result.isEmpty())
delegate.processFinish(result);
}
}
There is a for-loop in this Async Task.
for (int i = 0; i <= query.length-1; i = i+2)
And now finally I can explain my issue:
I usually use SELECT queries, sometimes I use an INSERT query (which can be done by a single query), but when I parse an Update Query, my for-loop stops iterating after the first pass, so i+2 never happens. The update queries look like this:
String updatequeries[] = {UPDATE delivery SET contactperson = 'Jon Doe' WHERE officeid = 5, " ", UPDATE delivery SET contactemail = 'abd#def.gh' WHERE officeid = 5, " "};
Why does this for loop stop running right after the first run? The debugger does not show anything unusual, everything was parsed right and there are no queries missing. Updating a table does not return any results, but nothing depends on result values here. I tried to run 20 update queries in a single string var, but the for loop stops after the first iteration anyway. No issues are displayed in the debugger or in the logs. Have I overseen something or is there anything I don't know? Might this be a bug? Please help me! This issue drives me crazy.

Navigation button is not working

I have read a similar post but I still cannot get what is the problem.
I created a table in ms access, named DOCTOR, there are columns: DoctorID(number), Name(text), PhoneNumber(number), Department(text) and Specialization(text)
I connect the database to java through UCanAccess, below is the code to get connection
import java.sql.*;
public class Doctor
{
public static Connection connection; //sharing the memory
public static Connection connect() throws ClassNotFoundException, SQLException
{
String db = "net.ucanaccess.jdbc.UcanaccessDriver";
Class.forName(db);
String url = "jdbc:ucanaccess://C:/Users/user.oemuser/Documents/Doctor.accdb";
connection = DriverManager.getConnection(url);
return connection;
}
}
In my GUI class, i have a method called getConnect to show the data from database to textfield
public void getConnect()
{
try
{
connection = Doctor.connect();
statement=connection.createStatement();
String sql = "SELECT * FROM DOCTOR";
results = statement.executeQuery(sql);
results.next();
id = results.getInt("DoctorID");
name = results.getString("DoctorName");
phone = results.getInt("PhoneNumber");
dept = results.getString("Department");
spec = results.getString("Specialization");
textField1.setText("" +id);
textField2.setText(name);
textField3.setText("" +nf3.format(phone));
textField4.setText(dept);
textField5.setText(spec);
}
catch(Exception ex)
{
ex.printStackTrace();
}
}
and below is the code for the button1 which is the next button.
if(evt.getSource() == button1)
{
try
{
connection = Doctor.connect();
connection.setAutoCommit(false);
statement=connection.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_UPDATABLE);
String sql1 = "SELECT * FROM DOCTOR";
results = statement.executeQuery(sql1);
if(results.next())
{
textField1.setText("" +results.getInt("DoctorID"));
textField2.setText(results.getString("DoctorName"));
textField3.setText("" +nf3.format(results.getInt("PhoneNumber")));
textField4.setText(results.getString("Department"));
textField5.setText(results.getString("Specialization"));
}
else
{
results.previous();
JOptionPane.showMessageDialog(null, "No more records");
}
connection.commit();
}
catch(Exception ex){
ex.printStackTrace();
}
}
Obviously the best component to use here is a JTable if you want to query all records within a particular database table or at the very least place the result set into an ArrayList mind you database tables can hold millions+ of records so memory consumption may be a concern. Now, I'm not saying that your specific table holds that much data (that's a lot of Doctors) but other tables might.
You can of course do what you're doing and display one record at a time but then you should really be querying your database for the same, one specific record at a time. You do this by modifying your SQL SELECT statement with the addition of the WHERE clause statement and playing off the ID for each database table record, something like this:
String sql = "SELECT * FROM DOCTOR WHERE DoctorID = " + number + ";";
But then again we need to keep in mind that, if the schema for your DoctorID field is set as Auto Indexed which of course allows the database to automatically place a incrementing numerical ID value into this field, the Index may not necessarily be in a uniform sequential order such as:
1, 2, 3, 4, 5, 6, 7, 8, 9, 10,.......
instead it could possibly be in this order:
1, 3, 4, 5, 6, 9, 10, 11, 16, 17,....
This sort of thing happens in MS Access Tables where a table record has been deleted. You would think that the ID slot that is deleted would be available to the next record added to the table and would therefore hold that removed ID value but that is not the case. The Auto Index Increment (autonumber) simply continues to supply increasing incremental values. There are of course ways to fix this sequencing mismatch but they are never a good idea and should truly be avoided since doing so can really mess up table relationships and other things within the database. Bottom line, before experimenting with your database always make a Backup of that database first.
So, to utilize a WHERE clause to play against valid record ID's we need to do something like this with our forward and reverse navigation buttons:
Your Forward (Next) Navigation Button:
if(evt.getSource() == nextButton) {
try {
connection = Doctor.connect();
connection.setAutoCommit(false);
number++;
long max = 0, min = 0;
ResultSet results;
Statement statement=connection.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_UPDATABLE);
// Get the minimum DoctorID value within the DOCTOR table.
String sql0 = "SELECT MIN(DoctorID) AS LowestID from DOCTOR";
results = statement.executeQuery(sql0);
while (results.next()){ min = results.getLong("LowestID"); }
// Get the maximum DoctorID value within the DOCTOR table.
sql0 = "SELECT MAX(DoctorID) AS HighestID from DOCTOR";
results = statement.executeQuery(sql0);
while (results.next()){ max = results.getLong("HighestID"); }
if (max <= 0) {
JOptionPane.showMessageDialog(null, "No records found in Doctor Table.");
return;
}
if (number > min) { previousButton.setEnabled(true); }
if (number > max) {
nextButton.setEnabled(false);
JOptionPane.showMessageDialog(null, "No more records");
number--;
}
results = null;
while (results == null) {
String sql1 = "SELECT * FROM DOCTOR WHERE DoctorID = " + number + ";";
results = statement.executeQuery(sql1);
long id = 0;
// Fill The GUI Form Fields....
while (results.next()){
//id = results.getLong("DoctorID");
textField1.setText("" +results.getInt("DoctorID"));
textField2.setText(results.getString("DoctorName"));
textField3.setText("" + results.getString("PhoneNumber"));
textField4.setText(results.getString("Department"));
textField5.setText(results.getString("Specialization"));
connection.commit();
return;
}
// ----------------------------------------------------------
if (id != number) { results = null; number++; }
}
}
catch(Exception ex){ ex.printStackTrace(); }
}
Your Reverse (Previous) Navigation Button:
if(evt.getSource() == previousButton) {
try {
connection = Doctor.connect();
connection.setAutoCommit(false);
number--;
long max = 0, min = 0;
ResultSet results;
Statement statement=connection.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_UPDATABLE);
// Get the minimum DoctorID value within the DOCTOR table.
String sql0 = "SELECT MIN(DoctorID) AS LowestID from DOCTOR";
results = statement.executeQuery(sql0);
while (results.next()){ min = results.getLong("LowestID"); }
// --------------------------------------------------------------------------
// Get the maximum DoctorID value within the DOCTOR table.
sql0 = "SELECT MAX(DoctorID) AS HighestID from DOCTOR";
results = statement.executeQuery(sql0);
while (results.next()){ max = results.getLong("HighestID"); }
// --------------------------------------------------------------------------
if (max <= 0) {
JOptionPane.showMessageDialog(null, "No records found in Doctor Table.");
return;
}
if (number < min) {
previousButton.setEnabled(false);
JOptionPane.showMessageDialog(null, "No more records");
number++;
}
if (number < max) { nextButton.setEnabled(true); }
results = null;
while (results == null) {
String sql1 = "SELECT * FROM DOCTOR WHERE DoctorID = " + number + ";";
results = statement.executeQuery(sql1);
long id = 0;
// Fill The GUI Form Fields....
while (results.next()){
textField1.setText("" +results.getInt("DoctorID"));
textField2.setText(results.getString("DoctorName"));
textField3.setText("" + results.getString("PhoneNumber"));
textField4.setText(results.getString("Department"));
textField5.setText(results.getString("Specialization"));
connection.commit();
return;
}
// ----------------------------------------------------------
if (id != number) { results = null; number--; }
}
}
catch(Exception ex){ ex.printStackTrace(); }
}
Things To DO...
So as to remove duplicate code, create a method named
getMinID() that returns a Long Integer data type. Allow this method to accept two String Arguments (fieldName and
tableName). Work the above code section used to gather the minimum DoctorID value within the DOCTOR table into the new
**getMinID() method. Use this new method to replace the formentioned code for both the Forward (Next) and Revese (Previous)
buttons.
So as to remove duplicate code, create a method named
getMaxID() that returns a Long Integer data type. Allow this method to accept two String Arguments (fieldName and
tableName). Work the above code section used to gather the maximum DoctorID value within the DOCTOR table into the new
getMaxID() method. Use this new method to replace the formentioned code for both the Forward (Next) and Revese (Previous)
buttons.
So as to remove duplicate code, create a void method named
fillFormFields(). Allow this method to accept two arguments, one as Connection (*connection) and another as ResultSet
(results) . Work the above code section used to Fill The GUI
Form Fields into the new fillFormFields() method. Use this new
method to replace the formentioned code for both the Forward (Next)
and Revese (Previous) buttons.
Things To Read That Might Be Helpful:
The SQL WHERE clause statement and the SQL ORDER BY statement for sorting your result set.
Searching For Records

Execute multiple SQL statements in java

I want to execute a query in Java.
I create a connection. Then I want to execute an INSERT statement, when done, the connection is closed but I want to execute some insert statement by a connection and when the loop is finished then closing connection.
What can I do ?
My sample code is :
public NewClass() throws SQLException {
try {
Class.forName("oracle.jdbc.driver.OracleDriver");
} catch (ClassNotFoundException e) {
System.out.println("Where is your Oracle JDBC Driver?");
return;
}
System.out.println("Oracle JDBC Driver Registered!");
Connection connection = null;
try {
connection = DriverManager.getConnection(
"jdbc:oracle:thin:#localhost:1521:orcl1", "test",
"oracle");
} catch (SQLException e) {
System.out.println("Connection Failed! Check output console");
return;
}
if (connection != null) {
Statement stmt = connection.createStatement();
ResultSet rs = stmt.executeQuery("SELECT * from test.special_columns");
while (rs.next()) {
this.ColName = rs.getNString("column_name");
this.script = "insert into test.alldata (colname) ( select " + ColName + " from test.alldata2 ) " ;
stmt.executeUpdate("" + script);
}
}
else {
System.out.println("Failed to make connection!");
}
}
When the select statement ("SELECT * from test.special_columns") is executed, the loop must be twice, but when (stmt.executeUpdate("" + script)) is executed and done, then closing the connection and return from the class.
Following example uses addBatch & executeBatch commands to execute multiple SQL commands simultaneously.
import java.sql.*;
public class jdbcConn {
public static void main(String[] args) throws Exception{
Class.forName("org.apache.derby.jdbc.ClientDriver");
Connection con = DriverManager.getConnection
("jdbc:derby://localhost:1527/testDb","name","pass");
Statement stmt = con.createStatement
(ResultSet.TYPE_SCROLL_SENSITIVE,
ResultSet.CONCUR_UPDATABLE);
String insertEmp1 = "insert into emp values
(10,'jay','trainee')";
String insertEmp2 = "insert into emp values
(11,'jayes','trainee')";
String insertEmp3 = "insert into emp values
(12,'shail','trainee')";
con.setAutoCommit(false);
stmt.addBatch(insertEmp1);
stmt.addBatch(insertEmp2);
stmt.addBatch(insertEmp3);
ResultSet rs = stmt.executeQuery("select * from emp");
rs.last();
System.out.println("rows before batch execution= "
+ rs.getRow());
stmt.executeBatch();
con.commit();
System.out.println("Batch executed");
rs = stmt.executeQuery("select * from emp");
rs.last();
System.out.println("rows after batch execution= "
+ rs.getRow());
}
}
Result:
The above code sample will produce the following result.The result may vary.
rows before batch execution= 6
Batch executed
rows after batch execution= = 9
Source: Execute multiple SQL statements
In the abscence of the schema or the data contained in each table I'm going to make the following assumptions:
The table special_columns could look like this:
column_name
-----------
column_1
column_2
column_3
The table alldata2 could look like this:
column_1 | column_2 | column_3
---------------------------------
value_1_1 | value_2_1 | value_3_1
value_1_2 | value_2_2 | value_3_2
The table alldata should, after inserts have, happened look like this:
colname
---------
value_1_1
value_1_2
value_2_1
value_2_2
value_3_1
value_3_2
Given these assumptions you can copy the data like this:
try (
Connection connection = DriverManager.getConnection("jdbc:oracle:thin:#localhost:1521:orcl1", "test", "oracle")
)
{
StringBuilder columnNames = new StringBuilder();
try (
Statement select = connection.createStatement();
ResultSet specialColumns = select.executeQuery("SELECT column_name FROM special_columns");
Statement insert = connection.createStatement()
)
{
while (specialColumns.next())
{
int batchSize = 0;
insert.addBatch("INSERT INTO alldata(colname) SELECT " + specialColumns.getString(1) + " FROM alldata2");
if (batchSize >= MAX_BATCH_SIZE)
{
insert.executeBatch();
batchSize = 0;
}
}
insert.executeBatch();
}
A couple of things to note:
MAX_BATCH_SIZE should be set to a value based on your database configuration and the data being inserted.
this code is using the Java 7 try-with-resources feature to ensure the database resources are released when they're finished with.
you haven't needed to do a Class.forName since the service provider mechanism was introduced as detailed in the JavaDoc for DriverManager.
There are two problems in your code. First you use the same Statement object (stmt) to execute the select query, and the insert. In JDBC, executing a statement will close the ResultSet of the previous execute on the same object.
In your code, you loop over the ResultSet and execute an insert for each row. However executing that statement will close the ResultSet and therefor on the next iteration the call to next() will throw an SQLException as the ResultSet is closed.
The solution is to use two Statement objects: one for the select and one for the insert. This will however not always work by default, as you are working in autoCommit (this is the default), and with auto commit, the execution of any statement will commit any previous transactions (which usually also closes the ResultSet, although this may differ between databases and JDBC drivers). You either need to disable auto commit, or create the result set as holdable over commit (unless that already is the default of your JDBC driver).

Efficient way to do batch INSERTS with JDBC

In my app I need to do a lot of INSERTS. Its a Java app and I am using plain JDBC to execute the queries. The DB being Oracle. I have enabled batching though, so it saves me network latencies to execute queries. But the queries execute serially as separate INSERTs:
insert into some_table (col1, col2) values (val1, val2)
insert into some_table (col1, col2) values (val3, val4)
insert into some_table (col1, col2) values (val5, val6)
I was wondering if the following form of INSERT might be more efficient:
insert into some_table (col1, col2) values (val1, val2), (val3, val4), (val5, val6)
i.e. collapsing multiple INSERTs into one.
Any other tips for making batch INSERTs faster?
This is a mix of the two previous answers:
PreparedStatement ps = c.prepareStatement("INSERT INTO employees VALUES (?, ?)");
ps.setString(1, "John");
ps.setString(2,"Doe");
ps.addBatch();
ps.clearParameters();
ps.setString(1, "Dave");
ps.setString(2,"Smith");
ps.addBatch();
ps.clearParameters();
int[] results = ps.executeBatch();
Though the question asks inserting efficiently to Oracle using JDBC, I'm currently playing with DB2 (On IBM mainframe), conceptually inserting would be similar so thought it might be helpful to see my metrics between
inserting one record at a time
inserting a batch of records (very efficient)
Here go the metrics
1) Inserting one record at a time
public void writeWithCompileQuery(int records) {
PreparedStatement statement;
try {
Connection connection = getDatabaseConnection();
connection.setAutoCommit(true);
String compiledQuery = "INSERT INTO TESTDB.EMPLOYEE(EMPNO, EMPNM, DEPT, RANK, USERNAME)" +
" VALUES" + "(?, ?, ?, ?, ?)";
statement = connection.prepareStatement(compiledQuery);
long start = System.currentTimeMillis();
for(int index = 1; index < records; index++) {
statement.setInt(1, index);
statement.setString(2, "emp number-"+index);
statement.setInt(3, index);
statement.setInt(4, index);
statement.setString(5, "username");
long startInternal = System.currentTimeMillis();
statement.executeUpdate();
System.out.println("each transaction time taken = " + (System.currentTimeMillis() - startInternal) + " ms");
}
long end = System.currentTimeMillis();
System.out.println("total time taken = " + (end - start) + " ms");
System.out.println("avg total time taken = " + (end - start)/ records + " ms");
statement.close();
connection.close();
} catch (SQLException ex) {
System.err.println("SQLException information");
while (ex != null) {
System.err.println("Error msg: " + ex.getMessage());
ex = ex.getNextException();
}
}
}
The metrics for 100 transactions :
each transaction time taken = 123 ms
each transaction time taken = 53 ms
each transaction time taken = 48 ms
each transaction time taken = 48 ms
each transaction time taken = 49 ms
each transaction time taken = 49 ms
...
..
.
each transaction time taken = 49 ms
each transaction time taken = 49 ms
total time taken = 4935 ms
avg total time taken = 49 ms
The first transaction is taking around 120-150ms which is for the query parse and then execution, the subsequent transactions are only taking around 50ms. (Which is still high, but my database is on a different server(I need to troubleshoot the network))
2) With insertion in a batch (efficient one) - achieved by preparedStatement.executeBatch()
public int[] writeInABatchWithCompiledQuery(int records) {
PreparedStatement preparedStatement;
try {
Connection connection = getDatabaseConnection();
connection.setAutoCommit(true);
String compiledQuery = "INSERT INTO TESTDB.EMPLOYEE(EMPNO, EMPNM, DEPT, RANK, USERNAME)" +
" VALUES" + "(?, ?, ?, ?, ?)";
preparedStatement = connection.prepareStatement(compiledQuery);
for(int index = 1; index <= records; index++) {
preparedStatement.setInt(1, index);
preparedStatement.setString(2, "empo number-"+index);
preparedStatement.setInt(3, index+100);
preparedStatement.setInt(4, index+200);
preparedStatement.setString(5, "usernames");
preparedStatement.addBatch();
}
long start = System.currentTimeMillis();
int[] inserted = preparedStatement.executeBatch();
long end = System.currentTimeMillis();
System.out.println("total time taken to insert the batch = " + (end - start) + " ms");
System.out.println("total time taken = " + (end - start)/records + " s");
preparedStatement.close();
connection.close();
return inserted;
} catch (SQLException ex) {
System.err.println("SQLException information");
while (ex != null) {
System.err.println("Error msg: " + ex.getMessage());
ex = ex.getNextException();
}
throw new RuntimeException("Error");
}
}
The metrics for a batch of 100 transactions is
total time taken to insert the batch = 127 ms
and for 1000 transactions
total time taken to insert the batch = 341 ms
So, making 100 transactions in ~5000ms (with one trxn at a time) is decreased to ~150ms (with a batch of 100 records).
NOTE - Ignore my network which is super slow, but the metrics values would be relative.
The Statement gives you the following option:
Statement stmt = con.createStatement();
stmt.addBatch("INSERT INTO employees VALUES (1000, 'Joe Jones')");
stmt.addBatch("INSERT INTO departments VALUES (260, 'Shoe')");
stmt.addBatch("INSERT INTO emp_dept VALUES (1000, 260)");
// submit a batch of update commands for execution
int[] updateCounts = stmt.executeBatch();
You'll have to benchmark, obviously, but over JDBC issuing multiple inserts will be much faster if you use a PreparedStatement rather than a Statement.
You can use this rewriteBatchedStatements parameter to make the batch insert even faster.
you can read here about the param: MySQL and JDBC with rewriteBatchedStatements=true
SQLite: The above answers are all correct. For SQLite, it is a little bit different. Nothing really helps, even to put it in a batch is (sometimes) not improving performance. In that case, try to disable auto-commit and commit by hand after you are done (Warning! When multiple connections write at the same time, you can clash with these operations)
// connect(), yourList and compiledQuery you have to implement/define beforehand
try (Connection conn = connect()) {
conn.setAutoCommit(false);
preparedStatement pstmt = conn.prepareStatement(compiledQuery);
for(Object o : yourList){
pstmt.setString(o.toString());
pstmt.executeUpdate();
pstmt.getGeneratedKeys(); //if you need the generated keys
}
pstmt.close();
conn.commit();
}
How about using the INSERT ALL statement ?
INSERT ALL
INTO table_name VALUES ()
INTO table_name VALUES ()
...
SELECT Statement;
I remember that the last select statement is mandatory in order to make this request succeed. Don't remember why though.
You might consider using PreparedStatement instead as well. lots of advantages !
Farid
You can use addBatch and executeBatch for batch insert in java See the Example : Batch Insert In Java
In my code I have no direct access to the 'preparedStatement' so I cannot use batch, I just pass it the query and a list of parameters. The trick however is to create a variable length insert statement, and a LinkedList of parameters. The effect is the same as the top example, with variable parameter input length.See below (error checking omitted).
Assuming 'myTable' has 3 updatable fields: f1, f2 and f3
String []args={"A","B","C", "X","Y","Z" }; // etc, input list of triplets
final String QUERY="INSERT INTO [myTable] (f1,f2,f3) values ";
LinkedList params=new LinkedList();
String comma="";
StringBuilder q=QUERY;
for(int nl=0; nl< args.length; nl+=3 ) { // args is a list of triplets values
params.add(args[nl]);
params.add(args[nl+1]);
params.add(args[nl+2]);
q.append(comma+"(?,?,?)");
comma=",";
}
int nr=insertIntoDB(q, params);
in my DBInterface class I have:
int insertIntoDB(String query, LinkedList <String>params) {
preparedUPDStmt = connectionSQL.prepareStatement(query);
int n=1;
for(String x:params) {
preparedUPDStmt.setString(n++, x);
}
int updates=preparedUPDStmt.executeUpdate();
return updates;
}
if you use jdbcTemplate then:
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.BatchPreparedStatementSetter;
public int[] batchInsert(List<Book> books) {
return this.jdbcTemplate.batchUpdate(
"insert into books (name, price) values(?,?)",
new BatchPreparedStatementSetter() {
public void setValues(PreparedStatement ps, int i) throws SQLException {
ps.setString(1, books.get(i).getName());
ps.setBigDecimal(2, books.get(i).getPrice());
}
public int getBatchSize() {
return books.size();
}
});
}
or with more advanced configuration
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.ParameterizedPreparedStatementSetter;
public int[][] batchInsert(List<Book> books, int batchSize) {
int[][] updateCounts = jdbcTemplate.batchUpdate(
"insert into books (name, price) values(?,?)",
books,
batchSize,
new ParameterizedPreparedStatementSetter<Book>() {
public void setValues(PreparedStatement ps, Book argument)
throws SQLException {
ps.setString(1, argument.getName());
ps.setBigDecimal(2, argument.getPrice());
}
});
return updateCounts;
}
link to source
Using PreparedStatements will be MUCH slower than Statements if you have low iterations. To gain a performance benefit from using a PrepareStatement over a statement, you need to be using it in a loop where iterations are at least 50 or higher.

Categories

Resources