Develop Reference

INSERT ALL - For more than 1000 rows

I have an INSERT ALL query in my program like:
<insert id="insertRecord" parameterType="java.util.List">
INSERT ALL
<foreach collection="myList" item="addrElement" index="index">
INTO MYTABLE (COLUMN1,COLUMN2,COLUMN3) values (#{addrElement.element1},#{addrElement.element2},#{addrElement.element3})
</foreach>
SELECT * FROM dual
</insert>
The list will hold a minimum of 10000 records.
Obviously, this is throwing an exception since INSERT ALL cannot handle more than 1000 records.
; bad SQL grammar []; nested exception is java.sql.SQLSyntaxErrorException: ORA-00913: too many values
I have checked many answers in SO as well as other sites to check that, picking records more than 1000 rows is specified only for SELECT query and not for an INSERT query.
Can someone lend me a hand on this? Would be much helpful.

You need to perform batch insert.
int batchSize = 100;
try (SqlSession sqlSession = sqlSessionFactory.openSession(ExecutorType.BATCH)) {
YourMapper mapper = sqlSession.getMapper(YourMapper.class);
int size = list.size();
for (int i = 0; i < size;) {
mapper.insertRecord(list.get(i));
i++;
if (i % batchSize == 0 || i == size) {
sqlSession.flushStatements();
sqlSession.clearCache();
}
}
sqlSession.commit();
}
You should find an appropriate value for the batchSize (it depends on various factors).
The insert statement is pretty straight forward.
<insert id="insertRecord">
INSERT INTO MYTABLE (COLUMN1, COLUMN2, COLUMN3)
VALUES (#{addrElement.element1}, #{addrElement.element2}, #{addrElement.element3})
</insert>
We have an FAQ entry.

Java- Insert records in batch does not work in MYSQL

So I have created a simple program to insert rows to my database(MYSQL) table. I am inserting the records in a batch of 1,000. It works perfectly fine but as is it inserting the 1st batch of 1,000 before completion, I am still able to check the records in my table and there exists records (<1000).
I thought inserting by batch means the behavior should be after the 1st 1000 records completed, then it starts populate the table and if I check the table before it completes the 1st batch, it should return an empty table. That's how I believe it should work.
Code(snippet):
for (int i=1; i < words.length; i++) {
preparedStatement.setString(1, path);
preparedStatement.setString(2, words[i]);
preparedStatement.addBatch();
if (i % 1000 == 0) {
preparedStatement.executeBatch();
System.out.print("Add Thousand");
}
}
if (words.length % 1000 > 0) {
preparedStatement.executeBatch();
System.out.print("Add Remaining");
}
Is this how the nature of MySQL behaves even though it is inserting in batches but it "appears" to be inserting row by row as it begins the batch?
Shouldn't the table appear empty until the first batch of 1000 has been executed?
Can anyone explain what is going on?

JDBC + Java - Batch updates doubts

I am using Oracle Database 12c with oracle.jdbc.driver.OracleDriver.
How batch insert is working? I know that is grouping statements, but what exactly if happening during preparedStatement.executeBatch? Is it executing only 1 insert per batch?
What approach for executing batches is better. This one with execute outside loop:
PreparedStatement ps = c.prepareStatement("insert into some_tab(id, val) values (?, ?);");
for (int i = 0; i < 3000; i++) {
ps.setLong(1, i);
ps.setString(2, "value" + i);
if ((i + 1) % 3 == 0) {
ps.addBatch();
}
}
ps.executeBatch();
Or this - with execution in loop:
PreparedStatement ps = c.prepareStatement("insert into some_tab(id, val) values (?, ?);");
for (int i = 0; i < 3000; i++) {
ps.setLong(1, i);
ps.setString(2, "value" + i);
ps.addBatch();
if ((i + 1) % 3 == 0) {
ps.executeBatch();
}
}
ps.executeBatch();

What approach for executing batches is better.
PreparedStatement.addBatch insert the current parameters into the batch
Adds a set of parameters to this PreparedStatement object's batch of commands.
preparedStatement.executeBatch send the current batch to the DB.
Submits a batch of commands to the database for execution
So your codes don't have the same logic. One will only add 1/3 of the insert queries into the batch. The other will execute the batch every 3 iteration.
I would sugget a mix of both, add each iteration into the batch, and every # iteration, execute it :
while(...){
...
ps.addBatch();
if ((i + 1) % 3 == 0) { // 3 is just the example, this can be much higher
ps.executeBatch();
}
}
//Send the rest if the loop ended with `(i + 1) % 3 != 0`
ps.executeBatch();
Note that a batch of 3 is probably not necessary, you can increase the value drastically, but I don't really know a way to "estimate" the size of a batch to be efficient, I usually use a batch of 1000 items but this is not to take for granted...
what exactly if happening during preparedStatement.executeBatch? Is it executing only 1 insert per batch?
I always see a batch like a package.
you write a label with the address (create PreparedStatement)
you take a box and stick the address on it
you fill the box with parameters (multiple addBatch)
if the box is too small, you send the current box (executeBatch)
if there is still items to send, restart at point 2
The idea is to limit the number of communication from the JDBC and the DB using a package/batch.
How this is working under the hood is not really answerable and should not really be a concerned.

Inserting huge data into a database in java

This may be asked a lot but,
so I am trying to insert a 4 million records to a database using java,
I did a lot of googling and tried access and MySQL. Both were almost the same,
with MySQL I tried statement.addBatch(); but still takes forever.
the question is, what is the best time I can get ? and what is the best way ?
counter++;
String sqlQuery = "INSERT INTO employees VALUES("some query")";
sqlState.addBatch(sqlQuery);
if (counter == 1000) {
sqlState.executeBatch();
counter = 0;
}
Also am I using the Batch right ?

Reuse a single PreparedStatements, and set parameters on it for each record, then add it to the batch.
PreparedStatement ps = conn.prepareStatement("Insert into employees (f1, f2, f3) VALUES (?, ?, ?)");
while (hasMoreRecords) {
ps.setString(1, "v1");
ps.setString(2, "v2");
....
ps.addBatch();
if (++i % 1000 == 0) {
ps.executeBatch();
}
}
ps.executeBatch();
This won't be a huge difference, but is optimal.
Does your INSERT use a sub-query? Maybe that sub-query is slow.

How to get number of rows from a resultResult?

I use the below approach to determine my result set is not empty and proceed to do assertions on the values.
...
resultSet = statement.executeQuery("select count(*) as rowCount from tbName where...");
while (resultSet.next()) {
rowCount = Integer.parseInt(resultSet.getString("rowCount"));
}
Assert.assertTrue(rowCount > 0);
resultSet = statement.executeQuery("select * from tbName where ...");
while (resultSet.next()) {
//do some assertions on values here.
}
...
Is there anyway to get the number of rows directly from the resultSet directly in a single query? Something like the below?
resultSet = statement.executeQuery("select * from tbName where ...");
if( resultSet.count/length/size > 0) {
}

You can change the query to include a column with the row count:
select t.*, count(*) over () as row_count
from tbName t
where ...
then you can get the count using
int rowCount rs.getInt("row_count");
Note that you won't get a 0 count because that means the actual query did not return anything in the first place. So you can't use that to verify if your query returned anything. If you only want to check if the result is empty, use next()
resultSet = statement.executeQuery(".....");
if (resultSet.next()) {
// at least one row returned
} else {
// no rows returned at all
}
Btw: you should always use the getXXX() method that matches the column's data type. Using getString() on all columns is not a good idea.

1) Moves the cursor to the last row: resultset.last();
2)Retrieves the current row number: int count = resultset.getRow();
Tips:
It's based on you create a statement via calling function "
Statement createStatement(int resultSetType,int resultSetConcurrency)
throws SQLException"
to gernerate a scrollable resultSet.

There are two ways to get number of rows.
1) if you want to check the number of rows exist in table you may use count query.
2) if you want to count number of rows in a result set you have to traverse that result set to count rows.