I have a result set which is not empty. It contains 6 columns. But if I want to use the loop, nothing happens.
If I call the stored procedure in SQL Server Management Studio with the same parameters as in the Java code, I got a result:
A few minutes ago everything worked
// That doesn't work because I can't get into the loop
if (con != null) {
String an_id = "bkoubik";
String AS_Aufruf = "exec BfV_Web.sp_Anwender_Start\n#AnwenderID = ?";
try {
PreparedStatement STMT = con.prepareStatement(AS_Aufruf);
STMT.setString(1, an_id);
ResultSet rs = STMT.executeQuery();
//THIS IF STATEMENT IS NOT ENTERING, SO WHY THEN THE WHILE LOOP
IS NOT WORKING ?
if (!rs.next() ) {
System.out.println("no data");
}
ResultSetMetaData meta = rs.getMetaData();
int intRS = meta.getColumnCount();
//THE LOOP IS NOT ENTERING
while (rs.next()) {
[...]
}
intRS is the column count, not the row count. Your rs is likely empty.
The first call to rs.next() (before the loop) already consumes the one and only row, so the next call (in the while) will return false.
void insert(ArrayList myList){
conn = openDBConnection(); //Database Connection
Iterator iterator = myList.iterator();
while(iterator.hasNext()){
insertIntoDB((myClass)iterator.next(),conn);
}
closeDBConnection();
}
void insertIntoDB(myClass myObject, Connection conn){
String query = "insert into myTable values(?,?)";
PreparedStatement myStatement = conn.prepareStatement(query);
myStatement.setInt(1,myObject.getMyKey());
myStatement.setInt(2,myObject.getMyValue());
myStatement.execute();
}
In the above code myList is an arrayList of objects with more than 1.2M entries. I am getting the following error after inserting around 1000 records:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at com.mysql.jdbc.PreparedStatement.<init>(PreparedStatement.java:437)
at com.mysql.jdbc.Connection.clientPrepareStatement(Connection.java:2187)
at com.mysql.jdbc.Connection.prepareStatement(Connection.java:4829)
at com.mysql.jdbc.Connection.prepareStatement(Connection.java:4734)
at com.att.research.space.SpaceDaoImpl.insertMapping(SpaceDaoImpl.java:99)
at com.att.research.space.ElementMappingLoader.insertMappingData(ElementMappingLoader.java:68)
at com.att.research.space.CorrelationEngine.loadMappingFiles(CorrelationEngine.java:69)
at com.att.research.space.CorrelationEngine.main(CorrelationEngine.java:25)
I tried using System.gc() inside the iterator loop. But I don't think it is a good way to code and also it is consuming lot of CPU cycle.
The above code is a sample code format of my original code.
You run out of memory because you did not close the statement. This is called a resource leak.
Using try-with-resources (Java 7+):
void insertIntoDB(myClass myObject, Connection conn){
String query = "insert into myTable values(?,?)";
try (PreparedStatement myStatement = conn.prepareStatement(query)) {
myStatement.setInt(1,myObject.getMyKey());
myStatement.setInt(2,myObject.getMyValue());
myStatement.execute();
}
}
Pre-Java 7:
void insertIntoDB(myClass myObject, Connection conn){
String query = "insert into myTable values(?,?)";
PreparedStatement myStatement = conn.prepareStatement(query);
try {
myStatement.setInt(1,myObject.getMyKey());
myStatement.setInt(2,myObject.getMyValue());
myStatement.execute();
} finally {
myStatement.close();
}
}
As others have suggested, you should re-use your statement, but it's the missing close() that's the main issue.
The whole point of PreparedStatement is to create it once and bind variables.
Here's how I'd suggest that you write it:
private static final String INSERT_QUERY = "insert into myTable values(?,?)";
public int insert(Connection c, List<MyClass> myList) {
int numRows = 0;
PreparedStatement ps = null;
try {
ps = c.prepareStatement(INSERT_QUERY);
for (MyClass x : myList) {
ps.setInt(1, x.getMyKey());
ps.setInt(2, x.getMyValue());
numRows += ps.executeUpdate();
}
} finally {
close(ps);
}
return numRows;
}
I left some details for you to figure out (e.g. that close method).
A word of advice: Lose that affection for naming everything "MyFoo". Your code is unreadable with such a naming convention. Think more carefully about how you name things.
move the statement out of the loop and user addBatch() and executeBatch instead
Avoiding memory leaks with finally block should definitely be addressed first; however if its just a very large resultset; I found this useful:
http://benjchristensen.com/2008/05/27/mysql-jdbc-memory-usage-on-large-resultset/
namely adding
stmt.setFetchSize(Integer.MIN_VALUE);
Before runing stmt.executeQuery();
I am extracting data from excel sheet and inserting them into my oracle table. The database is setup in a way that when executing a batch statement, if any insert statement in the batch fails, all the other statements in the batch are not executed. So my problem is how can I find out which row of data is actually causing it, so I can send a message to the user with the row number of the data that's causing the problem?
Connection con = null;
PreparedStatement pstmt = null;
Iterator iterator = list.iterator();
int rowCount = list.size();
int currentRow = 0;
String sqlStatement = "INSERT INTO DMD_VOL_UPLOAD (ORIGIN, DESTINATION, DAY_OF_WEEK, VOLUME)";
sqlStatement += " VALUES(?, ?, ?, ?)";
int batchSize==1000;
for(currentRow=1; currentRow<=rowCount; currentRow++){
ForecastBatch forecastBatch = (ForecastBatch) iterator.next();
pstmt.setString(1, forecastBatch.getOrigin());
pstmt.setString(2, forecastBatch.getDestination());
pstmt.setInt(3, forecastBatch.getDayOfWeek());
pstmt.setInt(4, forecastBatch.getVolumeSum());
pstmt.addBatch();
if(i % batchSize == 0){
updateCounts = pstmt.executeBatch();
con.commit();
pstmt.clearBatch();
session.flush();
session.clear();
}
}
executeBatch returns an integer array containing the counts of all the rows modified by each statement in the batch. I think negative numbers are used to indicate errors. You should be able to figure out which ones failed using this return value.
http://docs.oracle.com/javase/1.3/docs/guide/jdbc/spec2/jdbc2.1.frame6.html
java.sql.Statement's javadoc says that executeBatch() throws BatchUpdateException (a subclass of SQLException) if one of the commands sent to the database fails to execute properly or attempts to return a result set.
The method getUpdateCount() of java.sql.BatchUpdateException "..Retrieves the update count for each update statement in the batch update that executed successfully before this exception occurred..."
If none of this works, you will probably have to fall back to executing and committing each statement (within this particular batch) individually until you hit an error.
hi i am new to java and i am inserting in in to database using loop from array it takes time how would i insert data in DB as bulk insertion my code here,
if(con != null)
{
rs = dboperation.DBselectstatement(con,"select host_object_id from nagios_hosts where address='"+ip+"'");
if (rs != null)
{
rs.next();
String id = rs.getString(1);
for(int i= 0;i<serviceArray.length;i++)
{
status.append(serviceArray[i]+"\n");
dboperation.DbupdateStatement(DbAcess.getNagios_connection(),"insert into nagios_servicelist(service_name,host_object_id) values('"+serviceArray[i]+"','"+id+"')");
}
}
}
do not go in detail about this code i tell you that i am getting id from the first query in "rs" resultset and "servicearray" have services that i want to insert in Db but it takes time in loop how will i do this array bulk insertion in Database?
hopes to listen from you soon
Thanks in Advance
You shuld use JDBC bulk insert for your purpose -
//Create a new statement
Statement st = con.createStatement();
//Add SQL statements to be executed
st.addBatch("insert into nagios_servicelist(service_name,host_object_id) values('"+serviceArray[0]+"','"+id+"')");
st.addBatch("insert into nagios_servicelist(service_name,host_object_id) values('"+serviceArray[1]+"','"+id+"')");
st.addBatch("insert into nagios_servicelist(service_name,host_object_id) values('"+serviceArray[2]+"','"+id+"')");
// Execute the statements in batch
st.executeBatch();
You can insert your own logic here. But this is the overview of how this is to be done.
The following code avoids out of memory error as well as SQL injection
String sql = "insert into employee (name, city, phone) values (?, ?, ?)";
Connection connection = new getConnection();
PreparedStatement ps = connection.prepareStatement(sql);
final int batchSize = 1000;
int count = 0;
for (Employee employee: employees) {
ps.setString(1, employee.getName());
ps.setString(2, employee.getCity());
ps.setString(3, employee.getPhone());
ps.addBatch();
if(++count % batchSize == 0) {
ps.executeBatch();
}
}
ps.executeBatch(); // insert remaining records
ps.close();
connection.close();
In my app I need to do a lot of INSERTS. Its a Java app and I am using plain JDBC to execute the queries. The DB being Oracle. I have enabled batching though, so it saves me network latencies to execute queries. But the queries execute serially as separate INSERTs:
insert into some_table (col1, col2) values (val1, val2)
insert into some_table (col1, col2) values (val3, val4)
insert into some_table (col1, col2) values (val5, val6)
I was wondering if the following form of INSERT might be more efficient:
insert into some_table (col1, col2) values (val1, val2), (val3, val4), (val5, val6)
i.e. collapsing multiple INSERTs into one.
Any other tips for making batch INSERTs faster?
This is a mix of the two previous answers:
PreparedStatement ps = c.prepareStatement("INSERT INTO employees VALUES (?, ?)");
ps.setString(1, "John");
ps.setString(2,"Doe");
ps.addBatch();
ps.clearParameters();
ps.setString(1, "Dave");
ps.setString(2,"Smith");
ps.addBatch();
ps.clearParameters();
int[] results = ps.executeBatch();
Though the question asks inserting efficiently to Oracle using JDBC, I'm currently playing with DB2 (On IBM mainframe), conceptually inserting would be similar so thought it might be helpful to see my metrics between
inserting one record at a time
inserting a batch of records (very efficient)
Here go the metrics
1) Inserting one record at a time
public void writeWithCompileQuery(int records) {
PreparedStatement statement;
try {
Connection connection = getDatabaseConnection();
connection.setAutoCommit(true);
String compiledQuery = "INSERT INTO TESTDB.EMPLOYEE(EMPNO, EMPNM, DEPT, RANK, USERNAME)" +
" VALUES" + "(?, ?, ?, ?, ?)";
statement = connection.prepareStatement(compiledQuery);
long start = System.currentTimeMillis();
for(int index = 1; index < records; index++) {
statement.setInt(1, index);
statement.setString(2, "emp number-"+index);
statement.setInt(3, index);
statement.setInt(4, index);
statement.setString(5, "username");
long startInternal = System.currentTimeMillis();
statement.executeUpdate();
System.out.println("each transaction time taken = " + (System.currentTimeMillis() - startInternal) + " ms");
}
long end = System.currentTimeMillis();
System.out.println("total time taken = " + (end - start) + " ms");
System.out.println("avg total time taken = " + (end - start)/ records + " ms");
statement.close();
connection.close();
} catch (SQLException ex) {
System.err.println("SQLException information");
while (ex != null) {
System.err.println("Error msg: " + ex.getMessage());
ex = ex.getNextException();
}
}
}
The metrics for 100 transactions :
each transaction time taken = 123 ms
each transaction time taken = 53 ms
each transaction time taken = 48 ms
each transaction time taken = 48 ms
each transaction time taken = 49 ms
each transaction time taken = 49 ms
...
..
.
each transaction time taken = 49 ms
each transaction time taken = 49 ms
total time taken = 4935 ms
avg total time taken = 49 ms
The first transaction is taking around 120-150ms which is for the query parse and then execution, the subsequent transactions are only taking around 50ms. (Which is still high, but my database is on a different server(I need to troubleshoot the network))
2) With insertion in a batch (efficient one) - achieved by preparedStatement.executeBatch()
public int[] writeInABatchWithCompiledQuery(int records) {
PreparedStatement preparedStatement;
try {
Connection connection = getDatabaseConnection();
connection.setAutoCommit(true);
String compiledQuery = "INSERT INTO TESTDB.EMPLOYEE(EMPNO, EMPNM, DEPT, RANK, USERNAME)" +
" VALUES" + "(?, ?, ?, ?, ?)";
preparedStatement = connection.prepareStatement(compiledQuery);
for(int index = 1; index <= records; index++) {
preparedStatement.setInt(1, index);
preparedStatement.setString(2, "empo number-"+index);
preparedStatement.setInt(3, index+100);
preparedStatement.setInt(4, index+200);
preparedStatement.setString(5, "usernames");
preparedStatement.addBatch();
}
long start = System.currentTimeMillis();
int[] inserted = preparedStatement.executeBatch();
long end = System.currentTimeMillis();
System.out.println("total time taken to insert the batch = " + (end - start) + " ms");
System.out.println("total time taken = " + (end - start)/records + " s");
preparedStatement.close();
connection.close();
return inserted;
} catch (SQLException ex) {
System.err.println("SQLException information");
while (ex != null) {
System.err.println("Error msg: " + ex.getMessage());
ex = ex.getNextException();
}
throw new RuntimeException("Error");
}
}
The metrics for a batch of 100 transactions is
total time taken to insert the batch = 127 ms
and for 1000 transactions
total time taken to insert the batch = 341 ms
So, making 100 transactions in ~5000ms (with one trxn at a time) is decreased to ~150ms (with a batch of 100 records).
NOTE - Ignore my network which is super slow, but the metrics values would be relative.
The Statement gives you the following option:
Statement stmt = con.createStatement();
stmt.addBatch("INSERT INTO employees VALUES (1000, 'Joe Jones')");
stmt.addBatch("INSERT INTO departments VALUES (260, 'Shoe')");
stmt.addBatch("INSERT INTO emp_dept VALUES (1000, 260)");
// submit a batch of update commands for execution
int[] updateCounts = stmt.executeBatch();
You'll have to benchmark, obviously, but over JDBC issuing multiple inserts will be much faster if you use a PreparedStatement rather than a Statement.
You can use this rewriteBatchedStatements parameter to make the batch insert even faster.
you can read here about the param: MySQL and JDBC with rewriteBatchedStatements=true
SQLite: The above answers are all correct. For SQLite, it is a little bit different. Nothing really helps, even to put it in a batch is (sometimes) not improving performance. In that case, try to disable auto-commit and commit by hand after you are done (Warning! When multiple connections write at the same time, you can clash with these operations)
// connect(), yourList and compiledQuery you have to implement/define beforehand
try (Connection conn = connect()) {
conn.setAutoCommit(false);
preparedStatement pstmt = conn.prepareStatement(compiledQuery);
for(Object o : yourList){
pstmt.setString(o.toString());
pstmt.executeUpdate();
pstmt.getGeneratedKeys(); //if you need the generated keys
}
pstmt.close();
conn.commit();
}
How about using the INSERT ALL statement ?
INSERT ALL
INTO table_name VALUES ()
INTO table_name VALUES ()
...
SELECT Statement;
I remember that the last select statement is mandatory in order to make this request succeed. Don't remember why though.
You might consider using PreparedStatement instead as well. lots of advantages !
Farid
You can use addBatch and executeBatch for batch insert in java See the Example : Batch Insert In Java
In my code I have no direct access to the 'preparedStatement' so I cannot use batch, I just pass it the query and a list of parameters. The trick however is to create a variable length insert statement, and a LinkedList of parameters. The effect is the same as the top example, with variable parameter input length.See below (error checking omitted).
Assuming 'myTable' has 3 updatable fields: f1, f2 and f3
String []args={"A","B","C", "X","Y","Z" }; // etc, input list of triplets
final String QUERY="INSERT INTO [myTable] (f1,f2,f3) values ";
LinkedList params=new LinkedList();
String comma="";
StringBuilder q=QUERY;
for(int nl=0; nl< args.length; nl+=3 ) { // args is a list of triplets values
params.add(args[nl]);
params.add(args[nl+1]);
params.add(args[nl+2]);
q.append(comma+"(?,?,?)");
comma=",";
}
int nr=insertIntoDB(q, params);
in my DBInterface class I have:
int insertIntoDB(String query, LinkedList <String>params) {
preparedUPDStmt = connectionSQL.prepareStatement(query);
int n=1;
for(String x:params) {
preparedUPDStmt.setString(n++, x);
}
int updates=preparedUPDStmt.executeUpdate();
return updates;
}
if you use jdbcTemplate then:
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.BatchPreparedStatementSetter;
public int[] batchInsert(List<Book> books) {
return this.jdbcTemplate.batchUpdate(
"insert into books (name, price) values(?,?)",
new BatchPreparedStatementSetter() {
public void setValues(PreparedStatement ps, int i) throws SQLException {
ps.setString(1, books.get(i).getName());
ps.setBigDecimal(2, books.get(i).getPrice());
}
public int getBatchSize() {
return books.size();
}
});
}
or with more advanced configuration
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.ParameterizedPreparedStatementSetter;
public int[][] batchInsert(List<Book> books, int batchSize) {
int[][] updateCounts = jdbcTemplate.batchUpdate(
"insert into books (name, price) values(?,?)",
books,
batchSize,
new ParameterizedPreparedStatementSetter<Book>() {
public void setValues(PreparedStatement ps, Book argument)
throws SQLException {
ps.setString(1, argument.getName());
ps.setBigDecimal(2, argument.getPrice());
}
});
return updateCounts;
}
link to source
Using PreparedStatements will be MUCH slower than Statements if you have low iterations. To gain a performance benefit from using a PrepareStatement over a statement, you need to be using it in a loop where iterations are at least 50 or higher.