Develop Reference

out of memory when insert record batch through jdbc

I want to copy a table (10 million records) in originDB(sqlite3) into another database called targetDB.
The process of my method is:
read data from the origin table and generate a ResultSet, then generate corresponding insert sql about every record and execute commit to batch insert when the count of record reach 10000. The code as follow:
public void transfer() throws IOException, SQLException {
targetDBOperate.setCommit(false);//batch insert
int count = 0;
String[] cols = parser(propertyPath);//get fields of data table
String query = "select * from " + originTable;
ResultSet rs = originDBOperate.executeQuery(query);//get origin table
String base = "insert into " + targetTable;
while(rs.next()) {
count++;
String insertSql = buildInsertSql(base,rs,cols);//corresponding insert sql
targetDBOperate.executeSql(insertSql);
if(count%10000==0) {
targetDBOperate.commit();// batch insert
}
}
targetDBOperate.closeConnection();
}
The follow picture is the trend of using memory, and vertical axis represents memory usage
As we can say it will be bigger and bigger until out of memory. The stackoverflow has some relevant question such as Out of memory when inserting records in SQLite, FireDac, Delphi
, but I havent solve my problem for we use different implement method. My hypothesis is that when the count of record hasn't reach 10000, these corresponding insert sql will be cached in memory and they haven't been removed when execute commit by default? Every advice will be appreciate.

By moving a higher number of rows in SQLite or any other relational database you should follow some basic principles:
1) set autoCommit to false, i.e. do not commit each insert
2) use batch update, i.e. do not round trip for each row
3) use prepared statement, i.e. do not parse each insert.
Putting this together you get following code:
cn is the source connection, cn2 is the target connection.
For each inserted row you call addBatch, but only once per batchSize you call executeBatch which initiates a round trip.
Do not forget a last executeBatch at the end of the loop and the final commit.
cn2.setAutoCommit(false)
String SEL_STMT = "select id, col1,col2 from tab1"
String INS_STMT = "insert into tab2(id, col1,col2) values(?,?,?)"
def batchSize = 10000
def stmt = cn.prepareStatement(SEL_STMT)
def stmtIns = cn2.prepareStatement(INS_STMT)
rs = stmt.executeQuery()
while(rs.next())
{
stmtIns.setLong(1,rs.getLong(1))
stmtIns.setString(2,rs.getString(2))
stmtIns.setTimestamp(3,rs.getTimestamp(3))
stmtIns.addBatch();
i += 1
if (i == batchSize) {
def insRec = stmtIns.executeBatch();
i = 0
}
}
rs.close()
stmt.close()
def insRec = stmtIns.executeBatch();
stmtIns.close()
cn2.commit()
Sample test with your size with sqlite-jdbc-3.23.1:
inserted rows: 10000000
total time taken to insert the batch = 46848 ms
I do not observe any memory issues or problems with a large transaction

You are trying to fetch 10M records in one go by doing the following. This will definitely eat your memory like anything
String query = "select * from " + originTable;
ResultSet rs = originDBOperate.executeQuery(query);//get origin table
Use paginated queries to read the batches and do batch updates according.
You are not even doing a batch-update You are simply firing 10K queries one after the other by doing the following code
String insertSql = buildInsertSql(base,rs,cols);//corresponding insert sql
targetDBOperate.executeSql(insertSql);
if(count%10000==0) {
targetDBOperate.commit();// This simply means that you are commiting after 10K records
}

Mysql JDBC - not entering ResultSet + incorrect no. of rows

I came across a problem with going through a ResultSet I'm generating from my MySQL db. My query should return at most one row per table (I'm looping through several tables searching by employee number). I've entered data in some of the tables; but my test o/p says that the resultset contains 0 rows and doesn't go through the ResultSet at all. The o/p line it's supposed to print never appears. It was in a while loop before I realised that it'd be returning at most one row, at which point I just swapped the while(rs.next()) for an if(rs.first()). Still no luck. Any suggestions?
My code looks like this:
try
{
rsTablesList = stmt.executeQuery("show tables;");
while(rsTablesList.next())
{
String tableName = rsTablesList.getString(1);
//checking if that table is a non-event table; loop is skipped in such a case
if(tableName.equalsIgnoreCase("emp"))
{
System.out.println("NOT IN EMP");
continue;
}
System.out.println("i'm in " + tableName); //tells us which table we're in
int checkEmpno = Integer.parseInt(empNoLbl.getText()); //search key
Statement s = con.createStatement();
query = "select 'eventname','lastrenewaldate', 'expdate' from " + tableName + " where 'empno'=" + checkEmpno + ";"; // eventname,
System.out.println("query is \n\t" + query + "");
rsEventDetails = s.executeQuery(query) ;
System.out.println("query executed\n");
//next two lines for the number of rows
rsEventDetails.last();
System.out.println("no. of rows is " + rsEventDetails.getRow()+ "\n\n");
if(rsEventDetails.first())
{
System.out.println("inside the if");
// i will add the row now
System.out.println("i will add the row now");
// cdTableModel.addRow(new Object[] {evtname,lastRenewalDate,expiryDate});
}
}
}
My output looks like this:
I'm in crm
query is
select 'eventname','lastrenewaldate', 'expdate' from crm where 'empno'=17;
query executed
no. of rows is 0
I'm in dgr
query is
select 'eventname','lastrenewaldate', 'expdate' from dgr where 'empno'=17;
query executed
no. of rows is 0
NOT IN EMP
I'm in eng_prof
query is
select 'eventname','lastrenewaldate', 'expdate' from eng_prof where 'empno'=17;
query executed
no. of rows is 0
I'm in frtol
query is
select 'eventname','lastrenewaldate', 'expdate' from frtol where 'empno'=17;
query executed
no. of rows is 0
(and so on, upto 17 tables.)
The '17' in the query is the empno that I've pulled from the user.
The thing is that I've already entered data in the first two tables, crm and dgr. The same query in the command line interface works; this morning, I tried the program out and it returned data for the one table that had data in it (crm). The next time onwards, nothing.
Context: I'm working on a school project to create some software for my dad's office, it'll help them organise the training etc schedules for the employees. (a little like Google Calendar I guess.) I'm using Netbeans and Mysql on Linux Mint. There are about 17 tables in the database. The user selects an employee name and the program searches for all entries in the database that correspond to an 'event' (my generic name for a test/training/other required event) and puts them into a JTable.

The single quotes around the column names and table name in the creation of the query seem to have caused the problem. On changing them to backticks, retrieval works fine and the data comes in as expected.
Thank you, #juergend (especially for the nice explanation) and #nailgun!

JDBC Return generated key or existing key

I have a table with unique index to eliminate duplicates (simplified example)
CREATE TABLE `domain` (
`id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`subdomain` VARCHAR(255) NOT NULL,
`domain` VARCHAR(63) NOT NULL,
`zone` VARCHAR(63) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `UNIQUE` (`subdomain` ASC, `domain` ASC, `zone` ASC),
ENGINE = InnoDB;
I insert a lot of rows and i need to get primary keys returned (for other one-to-many inserts).
My problem is, that I insert a lot of duplicates and I need those keys returned too.
This is my solution which works, but isn't there more simple solution? With this I cannot use batch inserts and I want this to be most efficient.
PreparedStatement selectDomain = connection.prepareStatement("SELECT id FROM domain WHERE subdomain = ? AND domain = ? AND zone = ?");
PreparedStatement insertDomain = connection.prepareStatement("INSERT INTO domain(subdomain, domain, zone) VALUES (?,?,?)", Statement.RETURN_GENERATED_KEYS);
public int insertDomain(String subdomain, String domain, String zone) throws SQLException {
int domainId = 0;
selectDomain.setString(1, subdomain);
selectDomain.setString(2, domain);
selectDomain.setString(3, zone);
ResultSet resultSet = selectDomain.executeQuery();
if (resultSet.next()) {
domainId = resultSet.getInt(1);
} else {
insertDomain.setString(1, subdomain);
insertDomain.setString(2, subdomain);
insertDomain.setString(3, subdomain);
insertDomain.executeUpdate();
resultSet = insertDomain.getGeneratedKeys();
if (resultSet.next()) {
domainId = resultSet.getInt(1);
}
}
selectDomain.clearParameters();
insertDomain.clearParameters();
}

As I understand its not so easy approach for using batch execution. Your apporach is the best way to get the auto generated keys. There are few limitations of JDBC driver and it varies version to version, where getGeneratedKeys() works for single entry.
Please look into below links, it may help you :-
How to get generated keys from JDBC batch insert in Oracle?
http://docs.oracle.com/database/121/JJDBC/jdbcvers.htm#JJDBC28099

You could modify your INSERT to be something like this:
INSERT INTO domain (subdomain, domain, zone)
SELECT $subdomain, $domain, $zone
FROM domain
WHERE NOT EXISTS(
SELECT subdomain, domain, zone
FROM domain d
WHERE d.subdomain= $subdomain and d.domain=$domain and d.zone=$zone
)
LIMIT 1
Where $subdomain, $domain, $zone are the tag (properly quoted or as a placeholder of course) that you want to add if it isn't already there. This approach won't even trigger an INSERT (and the subsequent autoincrement wastage) if the tag is already there. You could probably come up with nicer SQL than that but the above should do the trick.
If your table is properly indexed then the extra SELECT for the existence check will be fast and the database is going to have to perform that check anyway.

How to insert a row into a that table has an increment id with CachedRowSet?

I'm trying to insert a row into a table that has an increment id with CachedRowSet(I'm using Java wit Java DB), but I got the following SQLException:
java.sql.SQLException: Failed on insert row
at...
SQLState: null
Error Code: 0
Message: Failed on insert row
Here's my code snippet:
private static String urlString = "jdbc:derby:testdb;create=true";
private static String userName = "testUser";
private static String password = "testPassword";
...
CachedRowSet crs = new CachedRowSetImpl();
crs.setUrl(urlString);
crs.setUsername(userName);
crs.setPassword(password);
crs.setCommand("SELECT * FROM test_table");
crs.execute();
crs.moveToInsertRow();
crs.updateString("str_value", "testValue");
crs.insertRow();
crs.moveToCurrentRow();
crs.acceptChanges();
The SQLException is thrown from crs.inertRow().
The CachedRowSet works well if the table does not have an auto increment id.
How can I successfully do this?
other details:
I use the following code to create my table:
conn = DriverManager.getConnection(urlString, userName, password);
Statement stmt = conn.createStatement();
stmt.execute("CREATE TABLE test_table("
+ "id INTEGER NOT NULL GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1),"
+ "str_value VARCHAR(20) NOT NULL,"
+ "CONSTRAINT primary_key PRIMARY KEY (id))");
System.out.println("Talbe test_table created.");

This is a duplicate of Failed on insert row using CachedRowSet.
Unfortunately, that happens because of the API definition:
Throws:
SQLException - if a database access error occurs; the result set concurrency is CONCUR_READ_ONLY, this method is called on a closed result set, if this method is called when the cursor is not on the insert row, or if not all of non-nullable columns in the insert row have been given a non-null value
You may be better off using JdbcRowSetImpl.
You can only stick with the CachedRowSet if you remove the NOT NULL constraint of the id column. Unfortunately, this would also mean to remove the PK constraint. I'd assume that is too big a sacrifice, but it's either that or no CachedRowSet.

try this
crs.updateNull(1);
crs.insertRow();
where 1 is your PK. Worcked for me in MySQL

How to insert record in many to many relationship link table

Current I am working on a hospital management system project. I have two tables doctor and patient and have made a link table Visits between them. Primary Keys of doctor and patient tables respectively are foreign keys of the Visits table. I am able to update the doctors table from my code. Whenever a record is inserted into the patients table, the doctor table is checked to find out whether the allocated doctor exists or not, if doctor exists and insertion is made into the visits table. For this I have created a trigger on Patients Table. Now whenever I try to register patients, the patients table gets updated but the code throws a MySQLException with message "Column Count Does Not Match Value Count At Row 1".
Code
this.stmt=this.mycon.createStatement();
String query_add_patient="insert into patients values ('"+nic+"','"+name+"','"+address+"','"+city+"',"+age+",'"+dob+"','"+telephone+"');";
String query_select_employees="select * from employees where employee_type='Doctor' and employee_qualification='"+doctor+"'";
ResultSet rs=this.stmt.executeQuery(query_select_employees);
if(!rs.next()) {
JOptionPane.showMessageDialog(this, "This Doctor is not available in this facility!!!","Admin Error",JOptionPane.ERROR_MESSAGE);
return false;
}
rs.first();
String pattern = "yyyy-m-d";
SimpleDateFormat format = new SimpleDateFormat(pattern);
String enic=rs.getString("employee_nic");
System.out.println(enic);
String query="insert into Visits (patient_nic,employee_nic) values ("+nic+","+enic+")";
this.stmt.executeUpdate(query_add_patient);
closeConnection();
openConnection();
this.stmt=mycon.createStatement();
this.stmt.executeUpdate(query);
return true;
Here the statement "insert into visits" causes the problem. I tried to close and open the connection again but that did not work. The structure for my visits table is
Visits
patient_nic (varchar(45)) FK --> patient.patient_nic
employee_nic (varchar(45)) FK --> employee-->employee_nic

Make Sure the Column Names Each Have a Matching Value and Vice Versa.
E.g
Good
$query = "INSERT INTO table (first_name, last_name)
VALUES(Mr,Kyaw)";
Bad
$query = "INSERT INTO table (first_name, last_name)
VALUES(Mr,Kyaw,Thu)"; //can give Column Count Does Not Match Value Count At Row 1 exception