Currently we are selecting data from one database and inserting it into a backup database(SQL SERVER).
This data always contains more than 15K records in one select.
We are using Enumeration to iterate over the data selected.
We are using JDBC PreparedStatement to insert data as:
Enumeration values = ht.elements(); -- ht is HashTable containing selected data.
while(values.hasMoreElements())
{
pstmt = conn.prepareStatement("insert query");
pstmt.executeUpdate();
}
I am not sure if this is the correct or efficient way to do the faster insert.
For inserting 10k rows it takes near about 30 min or more.
Is there any efficient way to make it fast?
Note: Not using any indexes on the table.
Use a batch insert, but commit after a few entris, don't try to send all 10K at once. Try investigating to get the best size, it' a trade off to memory vs network trips.
Connection connection = new getConnection();
Statement statement = connection.createStatement();
int i = 0;
for (String query : queries) {
statement.addBatch("insert query");
if ((i++ % 500) == 0) {
// Do an execute now and again, don't send too many at once
statement.executeBatch();
}
}
statement.executeBatch();
statement.close();
connection.close();
Also, from your code I'm not sure what you are doing, but use paramaterised queries rather than sending 10K insert statements as text. Something like:
String q= "INSERT INTO data_table (id) values (?)";
Connection connection = new getConnection();
PreparedStatement ps = connection.prepareStatement(q);
for (Data d: data) {
ps.setString(1, d.getId());
ps.addBatch();
}
ps.executeBatch();
ps.close();
connection.close();
You can insert all the values in one sql command:
INSERT INTO Table1 ( Column1, Column2 ) VALUES
( V1, V2 ), ( V3, V4 ), .......
You may also insert the values by bulks of 500 records, for example, if the query would become very big. It is not efficient at all to insert on row per statement remotely (using a connection). Another solution is to do the inserts using a stored procedure. You just pass the values to it as parameters.
Here is how you can do it using the INSERT command above:
Enumeration values = ht.elements(); -- ht is HashTable containing selected data.
int i=0;
String sql="";
while(values.hasMoreElements())
{
sql+="(" + values + ")"; //better use StringBuffer here
i++;
if(i % 500 == 0) {
pstmt = conn.prepareStatement("insert query "+sql);
pstmt.executeUpdate();
sql="";
}
else
sql += " , ";
}
Related
I use Oracle 11g, I found old code which should insert record to table
The code insert all values except NCLOB column and only if value exists (can be null) it then update the specific column (using PreparedStatement setCharacterStream()).
I want to change this code to one insert, but want to make sure there's no good reason to keep this flow - update after insert for specific NCLOB column?
Code (PreparedStatement ps and ps2):
ps = conn.prepareStatement(INSERT_RECORD);
//INSERT INTO DATA_RECORDS (ID) VALUES (?) and other values...
ps.setInt(1, id);
result = ps.executeUpdate() == 1;
if (data != null && result) {
StringReader reader = new StringReader(data);
ps2 = conn.prepareStatement(UPDATE_RECORD);
//UPDATE DATA_RECORDS SET DATA = ? WHERE ID = ?
ps2.setCharacterStream(1, reader, data.length());
ps.setInt(2, id);
...
ps2.executeUpdate();
I am using java to execute some SQL Queries. Some of them are Getting data from one database(A) and storing in a table in another database(B).After process is done i am deleting all data from table in database(B). I am repeating this process every 5 mins.
Code:
String sql = "delete from newtable";
stmt5 = conn.prepareStatement(sql);
stmt5.executeUpdate(sql);
String sql_1 = "select distinct tbl_alm_log_2000000000.Csn, tbl_alm_log_2000000000.IsCleared, tbl_alm_log_2000000000.Id,tbl_alm_log_2000000000.NEType, tbl_alm_log_2000000000.OccurTime, tbl_alm_log_2000000000.hostIP, tbl_alm_log_2000000000.ExtendInfo From fmdb.dbo.tbl_alm_log_2000000000 Where IsCleared = 0";
ResultSet rs = stmt_1.executeQuery(sql_1);
String sql_2 = "insert into newtable (CSN, IsCleared, Id, NEType, OccurTime, hostIP) values(?,?,?,?,?,?)";
conn.setAutoCommit(false);
PreparedStatement ps = conn.prepareStatement(sql_2);
final int batchSize = 1000;
int count = 0;
while (rs.next()){
ps.setString(1, rs.getString(1)); //csn
ps.setString(2, rs.getString(2)); //iscleared
ps.setString(3, rs.getString(3));//id
ps.setString(4, rs.getString(4));//netype
ps.setString(5, rs.getString(5));//occurtime
ps.setString(6, rs.getString(6));//hostip
ps.addBatch();
if(++count % batchSize == 0) {
ps.executeBatch();
}
}
ps.executeBatch(); // insert remaining records
conn.commit();
ps.close();
It runs perfectly for 10 -20 runs and then gives "duplicate entry error for "value" in Csn as it is Primary key".
I added Distinct keyword in query and it is still giving this error after 10-20 runs.
Note: I m deleting data from newtable befor start of process so it is always adding in a empty table.
Suggest where i am going wrong.
Looks like you have misunderstanding about how does distinct work. In query with several selected columns it will search for distinct tuples of values, not for distinct Csn column only.
There are different ways how to select distinct values by one column only. It generally depends on particular DBMS you use and logic you want to apply for multiply tuples found for same Csn column values. Consider for instance this question: DISTINCT for only one Column
One of general ideas: select distinct single values for Csn column only, then iterate through this list and select first tuple of values with this Csn value (I don't know is it suitable for you select first tuple or not).
when you insert the data , you can add if not exists not make sure your data is unique ( i considered CSN only column in PK)
if not exists(select 1 from tbl_alm_log_2000000000 where CSN=? )
insert into newtable (CSN, IsCleared, Id, NEType, OccurTime, hostIP) values(?,?,?,?,?,?)
pstmt = conn.prepareStatement(queryBuilder.toString());
pstmt.setString(count++, commonDTO.getGroupName());
pstmt.setString(count++, commonDTO.getSubGrpNameHindi());
pstmt.setString(count++, commonDTO.getGroupCode());
pstmt.setInt(count++, commonDTO.getIsActive());
pstmt.executeUpdate();
StringBuilder queryBuilder = new StringBuilder();
queryBuilder = queryBuilder.append(SLCMQueryConstant.Get_SubGrp_Id);
pstmt.setString (1, commonDTO.getGroupName());
rs = pstmt.executeQuery();
int x = 0;
while(rs.next())
{
x= rs.getInt("subject_Group_ID");
}
queryBuilder= queryBuilder.append(SLCMQueryConstant.MapSubject_to_SubGrp);
pstmt.setInt(count++, x);
pstmt.setInt(count++, commonDTO.getSubGrpID());
pstmt.setInt(count++, commonDTO.getSubjectMaxMark());
pstmt.setInt(count++, commonDTO.getSubjectMinMark());
pstmt.executeUpdate();
conn.commit();
I am trying to insert a new row into first table.then I want id of newly added
row and insert into 2nd table but there is an exception.
The error message tells you exactly what's wrong - executeQuery is for executing SELECT queries, not INSERTS/UPDATES. Use executeUpdate instead, as e.g. described here.
Seeing you have added your query above - to execute multiple queries, run multiple executeQuery / executeUpdate statements. As long as you don't have auto-commit set and commit the third query, all three queries will be executed as a single commit (i.e. cannot be interrupted).
It is not possible to run multiple queries with a single executeQuery / executeUpdate, separated by semicolon. This syntax is simply what many clients use to separate queries textually, but the clients will send these queries separately with multiple requests as well.
I'd guess that running three separate queries in a single transaction will be exactly what you need. However, if for some reason you absolutely want to do it with a single execute, you can use a stored procedure, which you prepare ahead of time.
You have to create prepared statements from your built queries:
pstmt = conn.prepareStatement(queryBuilder.toString());
pstmt.setString(count++, commonDTO.getGroupName());
pstmt.setString(count++, commonDTO.getSubGrpNameHindi());
pstmt.setString(count++, commonDTO.getGroupCode());
pstmt.setInt(count++, commonDTO.getIsActive());
pstmt.executeUpdate();
StringBuilder queryBuilder = new StringBuilder();
queryBuilder = queryBuilder.append(SLCMQueryConstant.Get_SubGrp_Id);
pstmt = conn.prepareStatement(queryBuilder.toString()); // <<<<<<-----
pstmt.setString(1, commonDTO.getGroupName());
Recordset rs = pstmt.executeQuery();
int x = 0;
while(rs.next())
{
x = rs.getInt("subject_Group_ID");
}
And you have to reset your prepared statement parameter counter:
queryBuilder= queryBuilder.append(SLCMQueryConstant.MapSubject_to_SubGrp);
pstmt = conn.prepareStatement(queryBuilder.toString()); // <<<<<<-----
count = 1; //<<<<<-----
pstmt.setInt(count++, x);
pstmt.setInt(count++, commonDTO.getSubGrpID());
pstmt.setInt(count++, commonDTO.getSubjectMaxMark());
pstmt.setInt(count++, commonDTO.getSubjectMinMark());
pstmt.executeUpdate();
conn.commit();
I have to **inserts 100s of rows** into the database such as all the column values are exactly same except one. For example sake consider a table
------------------
|UserId|Timestamp|
------------------
Now **only timestamp is changing for every insert**.
Is it advisable to use prepared statement in the following way ?
PreparedStatement pstmt = con.prepareStatement("INSERT INTO Pings (UserId,Timestamp) VALUES (?,?)";
pstmt.setInt(1,001); //setting user is
while(true){
pstmt.setTimestamp(2,getTimestamp());
pstmt.executeUpdate();
}
Compared to
while(true){
pstmt.setInt(1,001);
pstmt.setTimestamp(2,getTimestamp());
pstmt.executeUpdate();
}
Will first approach work given that I am setting 1st column value only once ?
I suggest you use batching with PreparedStatement.addBatch() and Statement.executeBatch(). That might look something like,
int count = 0;
while (true) {
pstmt.setInt(1, 001);
pstmt.setTimestamp(2, getTimestamp());
pstmt.addBatch();
if (++count % 50 == 0) { // <-- batch size of 50.
pstmt.executeBatch();
}
}
pstmt.executeBatch();
I have a managed bean which makes SQL queries to Oracle database. This is just very simple example how I make SQL queries. This is the table structure:
GLOBALSETTINGS
---------------------------------
SessionTTL VARCHAR2(40 BYTE)
MAXACTIVEUSERS NUMBER
ACTIVEUSERS VARCHAR2(20 BYTE)
I use this table just to store application settings. In the example listed below I can fetch just one string with one SQL statement. I want with SQL query to fetch the content of the three rows - SessionTTL, MAXACTIVEUSERS, ACTIVEUSERS. Is it possible?
public String CheckUserDB(String userToCheck) throws SQLException {
String storedPassword = null;
String SQL_Statement = null;
if (ds == null) throw new SQLException();
Connection conn = ds.getConnection();
if (conn == null) throw new SQLException();
try {
conn.setAutoCommit(false);
boolean committed = false;
try {
SQL_Statement = "SELECT Passwd from USERS WHERE Username = ?";
PreparedStatement passwordQuery = conn.prepareStatement(SQL_Statement);
passwordQuery.setString(1, userToCheck);
ResultSet result = passwordQuery.executeQuery();
if(result.next()){
storedPassword = result.getString("Passwd");
}
conn.commit();
committed = true;
} finally {
if (!committed) conn.rollback();
}
}
finally {
conn.close();
}
return storedPassword;
}
P.S I want the content of the rows.
I'm hoping I understand what you are asking for, but I fear I don't as it seems too simple, but anyway...
I think you want the contents of 3 columns, not rows. And yes you can, you just specify the columns you want returned in your SQL statement:
SELECT SessionTTL, MAXACTIVEUSERS, ACTIVEUSERS FROM GLOBALSETTINGS WHERE (condition)...
you can also use * as a shortcut for all columns iof you don't want to explicitly specify them:
SELECT * FROM GLOBALSETTINGS WHERE (condition)...
Some background reading on SQL syntax might be useful
If I read this correctly (sorry if mistaken), all you want to do is change your SQL command to select ALL COLUMNS in your database table.
To do so:
string SqlAll = #"SELECT Database.SessionTTL, Database.MAXACTIVEUSERS, Database.ACTIVEUSERS FROM Database";
This will retrieve ALL columns in the database. You can also have conditional statements in your queries when you want to filter for logical reasons, such as TOP 20 to get the first 20 results from the result set.
If you like to return multiple lines with one sql query, you may want to look into ArrayList as you need a loop, where the code would go through your records and match and find all possible results until the end of the records list.