I have an database import process where I have a list of files which correspond to tables. The files are simple csv text files filled with data.
I would like to speed up the data import by using bulk inserts, so I have the following method:
private void bulkInsert(String tableName) {
String sql = "INSERT INTO "+ tableName +" VALUES (?,?,?);";
SQLiteStatement statement = sampleDB.compileStatement(sql);
sampleDB.beginTransaction();
for (int i = 0; i<100; i++) {
statement.clearBindings();
statement.bindLong(1, i);
statement.bindLong(2, i);
statement.bindLong(3, i*i);
statement.execute();
}
sampleDB.setTransactionSuccessful();
sampleDB.endTransaction();
}
The above method is not the method I'm currently using. In the method. I have 3 columns of type long.
However, as I have different tables with columns of different data types, is it possible generalize the above method? Maybe iterate through the columns and get the type?
EDIT
Would bindAllArgsAsStrings work? The data stored in the columns might be text, integer or real. I might also have NULL values in the data.
Related
Here is a sample code.
sourceFileItemNo and entityDplStatus are array of some values.
Query query;
for(int i = 0 ;(sourceFileItemNo!=null && i< sourceFileItemNo.length); i++){
Object[] parameters = {entityDplStatus[i],"1",sourceFileItemNo[i]};
String queryString = "UPDATE GtcEntityDetailsValue c SET c.dplStatus=?1 where c.referenceNo=?2 and c.itemNo=?3";
query = manager.createQuery(queryString);
query = setQueryParameters(query, parameters);
query.executeUpdate();
}
Here in this case we are updating the details each time in each iteration. Does JPA provides provision to add the queries to a list or something and execute all the queries in a single shot just like old connection statement execute batch?
Yes you can easily do it using hibernate,pass the list of objects with updated values that you want to persist, then just do:
private void updteRecord(List<Records> records){
int batchSize=10;
for(int i=0;i<records.size();i++)
{
getSession().saveOrUpdate(records.get(i));
if(i%batchSize==0)
{
getSession().flush();
getSession().clear();
}
}
}
The List of records you are feeding to this method should contain the values you need to upodate.
I'm creating parcel machine program. Every parcel has unique parcelID which is exported to mysql db. The problem is that every time when I run the program, the program is counting parcelID from 0. I'm looking for a solution which will allow me to check the last parcelID in the database and create row after the last one.
Now it looks like this: 1. I'm creating a new row in db (successfully) by java program. 2. I'm closing the program after some time. 3. I run the program again and I can't add another new row because there is error "duplicate entry '1' for key 'PRIMARY'".
public static void post() throws Exception{
int parcelID = Parcel.generateID();
int clientMPNumber = Parcel.typeClientNumber();
int orderPassword = Parcel.generatePass();
try{
Connection con = getConnection();
PreparedStatement posted = con.prepareStatement("INSERT INTO Parcels.Orders (parcelID, clientMPNumber, orderPassword) VALUES ('"+parcelID+"', '"+clientMPNumber+"', '"+orderPassword+"')");
posted.executeUpdate();
}
catch(Exception e){
System.out.println(e);
}
finally{
System.out.println("Insert completed");
}
}
and the method is:
public static int generateID(){
parcelID = parcelID + 1;
return parcelID;
}
I'd let the database do the heavy lifting for you - Just define the parcelID column as serial instead of trying to set its value yourself.
You shouldn't use Id generation, just create auto_increment column in database table
As described here , define your primary key column to auto increment for each insert so your java code doesn't have to manually calculate primary key value each time.
If that is not a possibility, you need to show how you declare & initialize parcelID. As of your current code, parcelID looks to be a class level field that gets initialized to zero for each run so you always get the same value - 1. You need to initialize with last value from data base.
Also, implement suggestion as mentioned in comment to your question regarding PreparedStatement
There are a couple of things to attent to.
// parcelID should be an INT AUTOINCREMENT primary key.
try (PreparedStatement posted = con.prepareStatement(
"INSERT INTO Parcels.Orders (clientMPNumber, orderPassword) "
+ "VALUES (?, ?)",
Statement.RETURN_GENERATED_KEYS);
posted.setString(1, clientMPNumber);
posted.setString(2, orderPassword);
posted.executeUpdate();
try (ResultSet rsKey = posted.getGeneratedKeys()) {
if (rsKey.next()) {
int parcelID = rsKey.getInt(1);
return parcelID; // Or such
}try-with-resources
}
}
The database can deal with automatic numbering best, so that two transactions at the same time do not steal the same "next" number.
You should close things like Connection, PreparedStatement and ResultSet. This can best be done using the a bit awkward syntax of try-with-resources. That closes automatically even on exception and return.
PreparedStatements should be used with placeholders ?. This takes care for escaping special characters like ' in the password. Also prevents SQL injection.
Stylistic better use SQLException above Exception. Better maybe even a throws SQLException.
I am beginner in android development.
I have a question. what is ?s term used for in explanation below? i got it from the documentation of android developer.
public int update (String table, ContentValues values, String whereClause, String[] whereArgs)
Added in API level 1
Convenience method for updating rows in the database.
Parameters
table the table to update in
values a map from column names to new column values. null is a valid value that will be translated to NULL.
whereClause the optional WHERE clause to apply when updating. Passing null will update all rows.
whereArgs You may include ?s in the where clause, which will be replaced by the values from whereArgs. The values will be bound as Strings.
Returns
the number of rows affected
Basically its a variable to be filled in later. You should use these everywhere that data is coming from a user, a file, or anything else not hardcoded into the app. Why? Because it prevents security problems due to SQL injection. The variables cannot themselves be SQL, and will not be parsed as SQL by the database. So if all variables sent from users to the db are bind variables you remove that entire class of security issues from the app.
A PreparedStatement supports a mechanism called bind variables. For example,
SELECT * FROM table WHERE id = ?
In the above query, there is a single bind parameter for an id. You might use it (to get a row where id is 100) with something like
String sql = "SELECT * FROM table WHERE id = ?";
try (PreparedStatement ps = conn.prepareStatement(sql)) {
ps.setInt(1, 100);
try (ResultSet rs = ps.executeQuery()) {
if (rs.next()) {
}
}
} catch (SQLException e) {
e.printStackTrace();
}
Each ? corresponds, in order, to an index of the sql arguments passed in the method's last String[] whereArgs parameter
public int update (table, values, "age > ? AND age < ?", new String[] { "18", "25"});
The documentation meant literally using '?' in the whereClause statement. Simple example:
rawQuery("select * from todo where _id = ?", new String[] { id });
From the above statement, during execution, the ? will be replaced by the value of variable id.
This mechanism helps prevent SQL Injection.
I have to insert anywhere between 50 - 500 contact's information into the database. I have 4 arraylists that contain image, name, number, bool variable respectively.
Each row in the data is made up of a combination of all the 4 arraylists along with a SNO. Please refer to the image below.
My question is, let's say i have 500 contacts that I retrieve from the User's Contacts list. Is it a good thing that, I have a function that inserts each row at a time into the table and call it 500 times? or is there any other way? A mean idea would be to combine all the arraylists together, pass it to the function and retrieve the data there and repeat the insert statement 500 times.
What is better in terms of performance?
for(int i =0; i < 500; i++)
{
dbObj.insert_row(par1, par2, par3, par4, ...);
}
OR
function insert_row(Combined ArrayLists)
{
for(int i=0; i<500; i++)
{
db.execSql(//Insert Statement);
}
}
Insert data into Database - What is the best way to do it
I suggest you to create own object that will represent your table where properties of object will be equal to columns in table, e.q.
public class Contact {
private String name;
private String number;
private String image;
private boolean conn;
//getters and setters
}
Now your approach sounds like "spaghetti code". There is not need to have four ArrayLists and this design is not efficient and correct.
Now, you will have one ArrayList of Contact object with 500 childs.
What is the best way to insert?
For sure wrap your insertion to one TRANSACTION that speed up your inserts rapidly and what is the main your dealing with database becomes much more safer and then you don't need to care about possibility of losing database integrity.
Example of transaction(one method from my personal example project):
public boolean insertTransaction(int count) throws SQLException {
boolean result = false;
try {
db = openWrite(DataSource.getInstance(mContext));
ContentValues values = new ContentValues();
if (db != null) {
db.beginTransaction();
for (int i = 0; i < count; i++) {
values.put(SQLConstants.KEY_TYPE, "type" + i);
values.put(SQLConstants.KEY_DATE, new Date().toString());
db.insertOrThrow(SQLConstants.TEST_TABLE_NAME, SQLConstants.KEY_TYPE, values);
values.clear();
}
db.setTransactionSuccessful();
result = true;
}
return result;
}
finally {
if (db != null) {
db.endTransaction();
}
close(db);
}
}
If you are going to insert 500 records into database you should use transaction
database.beginTransaction();
try {
// perform inserts
database.setTransactionSuccessful();
finally {
database.endTranasction();
}
As mentioned before create Your own class to represent one row or use ContentValues class
SQlite doesn't provide possibility to insert many rows in one query like in MySQL but there is some way around it You can read here.
If You decide to use method described in this link it is better if You create a function to generate this query and run it just once. Otherwise As others mentioned already you may use transaction to improve a performance of many inserts.
Converting 4 arrays into one object array makes the code better. but you can create these objects without doing it like this.
Prepare the sql statement with bind variables (? or :vars), then execute the statement multiple times in a loop by setting the bind variables for each row.
String sql = "insert into..... values (?,?,?,?)";
// Get connection etc
PreparedStatement stmt = conn.prepareStatement(sql);
for(int i =0; i < 500; i++)
{
stmt.setString(1, name.get(i));
stmt.setNumber(2, number.get(i));
...
stmt.executeUpdate();
}
Read the Following Code:
public class selectTable {
public static ResultSet rSet;
public static int total=0;
public static ResultSet onLoad_Opetations(Connection Conn, int rownum,String sql)
{
int rowNum=rownum;
int totalrec=0;
try
{
Conn=ConnectionODBC.getConnection();
Statement stmt = Conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_READ_ONLY);
String sqlStmt = sql;
rSet = stmt.executeQuery(sqlStmt);
total = rSet.getRow();
}
catch(Exception e)
{
System.out.println(e.getMessage());
}
System.out.println("Total Number of Records="+totalrec);
return rSet;
}
}
The folowing code dos't show actual total:
total = rSet.getRow();
my jTable display 4 record in jTable but total = 0; when I evaluate through debug, it shows:
total=(int)0;
rather than total=(int)4
And if I use
rSet=last(); above from the code total = rSet.getRow();
then total shows accurate value = 4 but rSet return nothing. then jTable is empty.
Update me!
BalusC's answer is right! but I have to mention according to the user instance variable such as:
rSet.last();
total = rSet.getRow();
and then which you are missing
rSet.beforeFirst();
the remaining code is same you will get your desire result.
You need to call ResultSet#beforeFirst() to put the cursor back to before the first row before you return the ResultSet object. This way the user will be able to use next() the usual way.
resultSet.last();
rows = resultSet.getRow();
resultSet.beforeFirst();
return resultSet;
However, you have bigger problems with the code given as far. It's leaking DB resources and it is also not a proper OOP approach. Lookup the DAO pattern. Ultimately you'd like to end up as
public List<Operations> list() throws SQLException {
// Declare Connection, Statement, ResultSet, List<Operation>.
try {
// Use Connection, Statement, ResultSet.
while (resultSet.next()) {
// Add new Operation to list.
}
} finally {
// Close ResultSet, Statement, Connection.
}
return list;
}
This way the caller has just to use List#size() to know about the number of records.
The getRow() method retrieves the current row number, not the number of rows. So before starting to iterate over the ResultSet, getRow() returns 0.
To get the actual number of rows returned after executing your query, there is no free method: you are supposed to iterate over it.
Yet, if you really need to retrieve the total number of rows before processing them, you can:
ResultSet.last()
ResultSet.getRow() to get the total number of rows
ResultSet.beforeFirst()
Process the ResultSet normally
As others have answered there is no way to get the count of rows without iterating till the end. You could do that, but you may not want to, note the following points:
For a many RDBMS systems ResultSet is a streaming API, this means
that it does not load (or maybe even fetch) all the rows from the
database server. See this question on SO. By iterating to the
end of the ResultSet you may add significantly to the time taken to
execute in certain cases.
A default ResultSet object is not updatable and has a cursor
that moves forward only. I think this means that unless you
execute
the query with ResultSet.TYPE_SCROLL_INSENSITIVE rSet.beforeFirst() will throw
SQLException. The reason it is this way is because there is cost
with scrollable cursor. According to the documentation, it may throw SQLFeatureNotSupportedException even if you create a scrollable cursor.
Populating and returning a List<Operations> means that you will
also need extra memory. For very large resultsets this will not
work
at all.
So the big question is which RDBMS?. All in all I would suggest not logging the number of records.
One better way would be to use SELECT COUNT statement of SQL.
Just when you need the count of number of rows returned, execute another query returning the exact number of result of that query.
try
{
Conn=ConnectionODBC.getConnection();
Statement stmt = Conn.createStatement();
String sqlStmt = sql;
String sqlrow = SELECT COUNT(*) from (sql) rowquery;
String total = stmt.executeQuery(sqlrow);
int rowcount = total.getInt(1);
}
The getRow() method will always yield 0 after a query:
ResultSet.getRow()
Retrieves the current row number.
Second, you output totalrec but never assign anything to it.
You can't get the number of rows returned in a ResultSet without iterating through it. And why would you return a ResultSet without iterating through it? There'd be no point in executing the query in the first place.
A better solution would be to separate persistence from view. Create a separate Data Access Object that handles all the database queries for you. Let it get the values to be displayed in the JTable, load them into a data structure, and then return it to the UI for display. The UI will have all the information it needs then.
I have solved that problem. The only I do is:
private int num_rows;
And then in your method using the resultset put this code
while (this.rs.next())
{
this.num_rows++;
}
That's all
The best way to get number of rows from resultset is using count function query for database access and then rs.getInt(1) method to get number of rows.
from my code look it:
String query = "SELECT COUNT() FROM table";
ResultSet rs = new DatabaseConnection().selectData(query);
rs.getInt(1);
this will return int value number of rows fetched from database.
Here DatabaseConnection().selectData() is my code for accessing database.
I was also stuck here but then solved...