I am trying to do Batch Insert for a table in Informix.
I tried following code to perform Batch Insert for a normal table.
PreparedStatement ps = conn.prepareStatement("insert into tableName (a,b,c,d) values(?,?,?,?)");
ps.addBatch();
int[] n = ps.executeBatch();
System.out.println("Array size : " + n.length); // Output : Array size : 1
And the above code works successfully for normal table.
But when I try to do the same batch insert on time-series table, its not working, neither there is any Exception caught, also the return type int[] of executeBatch() gives me count as zero.
int[] n = ps.executeBatch();
System.out.println("Array size : " + n.length); // Output : Array size : 0
Any idea what am I missing or doing wrong?
One thing to remember is that for Informix, Timeseries looks a lot like a table inside of a table. So when you insert into your base table, you have a column of type "timeseries". The actual timeseries data then goes into that column. This leads to some strange looking SQL as you have to run "UPDATE" statements to "insert" timeseries data since we are basically manipulating that one column in that one row in our base table.
Here is a full example that uses JDBC to setup a basic timeseries table and do batched inserts on a timeseries column. This example presumes you enter the timeseries data every minute and simulates that.
try(Connection c = DriverManager.getConnection("jdbc:informix-sqli://HOST:PORT/DATABASENAME", "username", "password") {
try(Statement s = c.createStatement()) {
//Auto registers timeseries if it does not exist (12.10 or higher versions of the server I believe)
s.execute("INSERT INTO CalendarPatterns VALUES ('patt_1min', '{1 on , 59 off}, second')");
s.execute("INSERT INTO CalendarTable (c_name, c_calendar)" +
" VALUES ('cal_1min', 'startdate(2018-01-01 00:00:00), " +
" pattstart(2018-01-01 00:00:00), pattname(patt_1min)')");
s.execute("CREATE ROW TYPE ts_basic_row(entry_time DATETIME YEAR TO FRACTION(5), value float NOT NULL)");
s.execute("CREATE TABLE tstab1( id integer, sensor timeseries(ts_basic_row))");
s.execute("EXECUTE PROCEDURE TSContainerCreate ('test_container', 'rootdbs','ts_basic_row', 0, 0)");
}
//Insert a row with a timeseries column
//Note the origin date matches the calendar pattern and calendar from above (explaining those is another exercise)
try(PreparedStatement p = c.prepareStatement("INSERT INTO tstab1 VALUES(?, ?)")) {
p.setInt(1, 1);
p.setString(2, "origin(2018-01-01 12:00:00.00000), calendar(cal_1min), container(test_container), threshold(0), irregular, []");
p.execute();
}
//Now we can bulk insert into our timeseries
//There are other mechanisms (setting IFX_USEPUT, using a bulk loader, etc) which could be faster, but this is a good start
Calendar cal = Calendar.getInstance();
Random r = new Random();
try(PreparedStatement p = c.prepareStatement("UPDATE tstab1 SET sensor = PutElem(sensor, ROW(?, ?)::ts_basic_row) WHERE id=?")) {
for(int i = 0; i < 1000; i++) {
p.setDate(1, new java.sql.Date(cal.getTimeInMillis()));
//add a minute to the calendar
cal.add(Calendar.MINUTE, 1);
p.setDouble(2, r.nextDouble()); //your sensor/timeseries value
p.setInt(3, 1); //The row in your base table (matching the id column)
p.addBatch();
}
int [] results = p.executeBatch();
System.out.println(Arrays.toString(results)); // a bunch of '1' values as expected
}
}
Related
In Android/Java, I am trying to compute and store the difference between 2 values in sqlite when one value is entered.
My table looks like this:
When weights are added/stored in the table in the column 'Weight', the column 'Diff_Weight' shall receive "Weight(N) - Weight(N-1)". Example: the last cell of Diff_Weight (row 6) = 88.0 - 55.2 = 32.8. // Row 5 shall get '-0.7' etc. Their type is REAL (col Weight & col Diff_Weight).
This 32.8 should be calculated and added at the same time when 88.0 is added to the table.
So far, I have read lots of tutorials and can't figure how to proceed. (My code to create and insert in the DB is fine, but reading is somehow more complex).
My code to read the entry is very bad because I don't see how to set it up:
public Bouble getData() {
String selectQuery= "SELECT * FROM " + TABLE_NAME2 + " ORDER BY COL_4 DESC LIMIT 1";
SQLiteDatabase db = this.getWritableDatabase();
Cursor cursor = db.rawQuery(TABLE_NAME2, null);
result2 = Double.valueOf(cursor.getString(cursor.getColumnIndex("Weight")));
result1 = Double.valueOf(cursor.getString(cursor.getColumnIndex("Weight")-1));
insertdata(result2-result1); //insert in row 6 of Diff_Weight
return
}
Can anybody help there?
If that is unclear, I was needing some help for the sqlite command AND the java to get the difference result.
Simplistically you can get the data by joining to the same table
SELECT a.id, a.weight, b.weight, (b.weight - a.weight) FROM TABLE_NAME2 a
join TABLE_NAME2 b on (b.id = a.id + 1);
One way is to use the lag() window function to get the value of the previous row (As ordered by id; using timestamps would be better but between splitting up the date and time into different columns and not using a date format that can be meaningfully sorted, this is easier.):
SELECT id, weight,
round(coalesce(weight - lag(weight, 1) OVER (ORDER BY id), weight), 1) AS diff_weight
FROM example
ORDER BY id
which gives
id weight diff_weight
---------- ---------- -----------
1 22.0 22.0
2 22.2 0.2
3 55.0 32.8
4 55.9 0.9
5 55.2 -0.7
6 88.0 32.8
You can make a view of this query use that like a normal table if you like. Generating the differences dynamically like this has the advantage that if an existing weight value changes, everything that depends on it doesn't have to be updated.
ok, after a long search, here is a possible result (sqlite + java):
first, you need to query the last row of the table...
...and handle the case if there is no row in your table (blank or new table)
then you must query the 'Weight' value from the known column 'Weight' (Y) and the row with ID you already have (X)
and when you have your value (last_weight), you need to write the difference (weight-last_weight) in the column 'Diff_Weight'.
Here is the code:
SQLiteDatabase db = this.getWritableDatabase();
//query the last row of the table
Cursor cursor = db.rawQuery("SELECT ID FROM TABLE_NAME2 ORDER BY ID DESC LIMIT 1", null);
cursor.moveToLast();
int lastID;
//handle the case if there is no row in your table
try {
lastID = cursor.getInt(0);
} catch (Exception e) {
lastID = 0;
}
Double lastWeight = 0.0;
//query the 'Weight' value
if (lastID >= 1) {
Cursor cursor2 = db.rawQuery("SELECT Weight FROM TABLE_NAME2 WHERE ID=" + lastID, null);
if (cursor2.moveToFirst()) { //this is boundary otherwise 'lastWeight' doesn't get the value
lastWeight= cursor2.getDouble(0);
}
} else {
lastWeight = 0.0;
}
//write the difference in the Diff_Weight column (=COL_5)
ContentValues cValues2 = new ContentValues();
//add your data in COL_1 to COL_4 here...
cValues2.put(COL_5, weight - lastWeight);
long id2 = db.insert(TABLE_NAME2, null, cValues2);
... And so you get the red figures in the column Diff_Weight from the table photo in the question.
I have data in following format:
HashMap<PageID, Set<SubscriberIDS>>
What I need to check is how many SubscriberIDS for each of the PageIDs do not exist in a MySQL table already. MySQL table has PageID and SubscriberID columns.
This is what I have so far:
String NEW_SUBSCRIBER_COUNT = "SELECT ? - COUNT(*) as new_subscribers from FB_SUBSCRIPTIONS WHERE PAGEID=? AND SUBSCRIBERID IN (?)";
First parameter being numberOFSubscriberIDs, Second being PageId and Third being SubscriberIds
but this will need to be hit for each pageId. How do I modify it to give me number of new subscribers for each PageID using single query.
Is there any specific need to do it in one query? Because while it can, it might actually be more readable to use your original solution and invoke a query for each page id. In any case, what you want can't be done in a single line, so you need to expect to loop at a given point.
// Build dynamic query
StringBuilder whereClause = new StringBuilder();
Iterator<PageID> it = yourMap.keySet().iterator();
while(it.hasNext()){
PageID key = it.next();
Set<SubscriberIDS> value = yourMap.get(key);
// You need to fill the 'IN' clause with multiple parameters, one for each subscriber id
StringBuilder inClause = new StringBuilder();
for(SubscriberIDS subId : value){
if(inClause.length > 0){
inClause.append(", ");
}
inClause.append("?");
preparedStatement.setInt(paramIndex++, subId.getId());
}
// For each page id we append a new 'OR' to our query
if(whereClause.lenght > 0){
whereClause.append(" OR ");
}
whereClause.append("(PAGEID=? AND SUBSCRIBERID IN (").append(inClause.toString()).append("))");
}
String query = "SELECT PAGEID, COUNT(SUBSCRIBERID) AS SUBSCRIBERS FROM FB_SUBSCRIPTIONS WHERE " + whereClause.toString() + " GROUP BY PAGEID";
// Create prepared statement and set parameters
PreparedStatement preparedStatement = connection.prepareStatement(query);
int paramIndex = 0;
it = yourMap.keySet().iterator();
while(it.hasNext()){
PageID key = it.next();
Set<SubscriberIDS> value = yourMap.get(key);
preparedStatement.setInt(paramIndex++, key.getId());
for(SubscriberIDS subId : value){
preparedStatement.setInt(paramIndex++, subId.getId());
}
}
// Execute query, loop over result and calculate new subscriptions
ResultSet rs = preparedStatement.executeQuery();
while(rs.next()){
int pageId = rs.getInt("PAGEID");
int newSubscriptions = yourMap.get(pageId).size() - rs.getInt("SUBSCRIBERS");
System.out.println(pageId + ", " + newSubscriptions);
}
Given following data in your map:
PAGEID SUBSCRIBERIDS
1 1,3,4,5,9
2 3,4,5,6,8,9
3 2,5,6
And following data in the DB:
PAGEID SUBSCRIBERIDS
1 3,4,10,11
2 1,2,5,7
3 1,2,5,6,7,8,9
This should give following output:
1,3
2,6
3,0
I haven't actually ran the code, so it might need some adjustments, but it gives you the general idea...
I want to copy a table (10 million records) in originDB(sqlite3) into another database called targetDB.
The process of my method is:
read data from the origin table and generate a ResultSet, then generate corresponding insert sql about every record and execute commit to batch insert when the count of record reach 10000. The code as follow:
public void transfer() throws IOException, SQLException {
targetDBOperate.setCommit(false);//batch insert
int count = 0;
String[] cols = parser(propertyPath);//get fields of data table
String query = "select * from " + originTable;
ResultSet rs = originDBOperate.executeQuery(query);//get origin table
String base = "insert into " + targetTable;
while(rs.next()) {
count++;
String insertSql = buildInsertSql(base,rs,cols);//corresponding insert sql
targetDBOperate.executeSql(insertSql);
if(count%10000==0) {
targetDBOperate.commit();// batch insert
}
}
targetDBOperate.closeConnection();
}
The follow picture is the trend of using memory, and vertical axis represents memory usage
As we can say it will be bigger and bigger until out of memory. The stackoverflow has some relevant question such as Out of memory when inserting records in SQLite, FireDac, Delphi
, but I havent solve my problem for we use different implement method. My hypothesis is that when the count of record hasn't reach 10000, these corresponding insert sql will be cached in memory and they haven't been removed when execute commit by default? Every advice will be appreciate.
By moving a higher number of rows in SQLite or any other relational database you should follow some basic principles:
1) set autoCommit to false, i.e. do not commit each insert
2) use batch update, i.e. do not round trip for each row
3) use prepared statement, i.e. do not parse each insert.
Putting this together you get following code:
cn is the source connection, cn2 is the target connection.
For each inserted row you call addBatch, but only once per batchSize you call executeBatch which initiates a round trip.
Do not forget a last executeBatch at the end of the loop and the final commit.
cn2.setAutoCommit(false)
String SEL_STMT = "select id, col1,col2 from tab1"
String INS_STMT = "insert into tab2(id, col1,col2) values(?,?,?)"
def batchSize = 10000
def stmt = cn.prepareStatement(SEL_STMT)
def stmtIns = cn2.prepareStatement(INS_STMT)
rs = stmt.executeQuery()
while(rs.next())
{
stmtIns.setLong(1,rs.getLong(1))
stmtIns.setString(2,rs.getString(2))
stmtIns.setTimestamp(3,rs.getTimestamp(3))
stmtIns.addBatch();
i += 1
if (i == batchSize) {
def insRec = stmtIns.executeBatch();
i = 0
}
}
rs.close()
stmt.close()
def insRec = stmtIns.executeBatch();
stmtIns.close()
cn2.commit()
Sample test with your size with sqlite-jdbc-3.23.1:
inserted rows: 10000000
total time taken to insert the batch = 46848 ms
I do not observe any memory issues or problems with a large transaction
You are trying to fetch 10M records in one go by doing the following. This will definitely eat your memory like anything
String query = "select * from " + originTable;
ResultSet rs = originDBOperate.executeQuery(query);//get origin table
Use paginated queries to read the batches and do batch updates according.
You are not even doing a batch-update You are simply firing 10K queries one after the other by doing the following code
String insertSql = buildInsertSql(base,rs,cols);//corresponding insert sql
targetDBOperate.executeSql(insertSql);
if(count%10000==0) {
targetDBOperate.commit();// This simply means that you are commiting after 10K records
}
I am trying to fetch records from oracle db. I have a select query which has an order by clause implemented. Now when I fire the query on the toad i get the results in correct order. i.e the order of the records at 10:00 AM is like
Record 1, Record 2,Record 3 and at 10:05 its Record 1, Record 2, Record 3. This is what i need.
Now when iam fetching it through java code, JDBC . I try to iterate the resultset, but here at 10:05 am I am getting the order like Record 2, Record 1, Record 3. Due to this when i am adding the records to the arraylist the order is not mantained.
I dont want to sort the records of arraylist after adding.
Can someone please let me know why using jdbc the records are not fetched in the order we can see using toad ?
Sample code
try{
List<TestObjVO> testResults = new ArrayList<TestObjVO>();
double statusValue = 0;
//Connection code
pstmt = conn.prepareStatement(QUERY);
ResultSet rs = pstmt.executeQuery();
while (rs.next()) {
testObj = new TestObjVO();
String date = rs.getString(1);
String status = rs.getString(2);
String svc_nm= rs.getString(3);
if("SUCCESS".equalsIgnoreCase(status)){
statusValue = 1;
}else if("ERROR".equalsIgnoreCase(status)){
statusValue = -1;
}
testObj.setTime(date);
testObj.setStatus(statusValue);
testObj.setSvc_nm(svc_nm);
testResults.add(testObj);
}
SELECT query
SELECT to_char(PROBING_DATE,'DD-MM-YYYY HH24:MI:SS') AS PROBING_DATE, STATUS, SERVICE_NAME FROM TABLE_NAME WHERE PROBING_DATE >= (sysdate-30/1440) ORDER BY PROBING_DATE,SERVICE_NAME
Table
create table TABLE_NAME(
probing_date TIMESTAMP(6) not null,
status VARCHAR2(8) not null,
service_name VARCHAR2(128) not null
)
Change your select to something like this:
SELECT to_char(PROBING_DATE,'DD-MM-YYYY HH24:MI:SS') AS PROBING_DATE_STR,
PROBING_DATE,
STATUS,
SERVICE_NAME
FROM TABLE_NAME
WHERE PROBING_DATE >= (sysdate-30/1440)
ORDER BY PROBING_DATE,SERVICE_NAME;
Note there's an extra field returned and is the raw TIMESTAMP field.
I only need the two most current items in a result set and was wondering what the best way to do that would be without a break. I realize that rs.next() returns true or false and tried to stop it with a counter but that failed. I have this at the moment:
while(rs.next()){
String name = rs.getString("name");
String startTime = rs.getString("starting_time");
String endTime = rs.getString("ending_time");
String date = rs.getString("directory");
String loc = rs.getString("location");
htmlBuilder.append("<li><a href='public/committees/calendar'>"+ name+"<br>");
htmlBuilder.append(date +" "+startTime+" - "+endTime+"</a> <!-- Link/title/date/start-end time --><br>");
htmlBuilder.append("<strong>Location: </strong>"+loc+"<br>");
htmlBuilder.append("</li>");
}
html = htmlBuilder.toString();
As you can tell, this returns everything from the ResultSet but I only need the first two entries.
Here is my correct query:
SELECT to_char(to_date(to_char(x.starting_date), 'J'),'mm/dd/yyyy') as start_date, to_char(to_date(to_char(x.ending_date), 'J'),'mm/dd/yyyy') as end_date, to_char(to_date(to_char(x.starting_date), 'J'),'yyyy-mm-dd') as directory, x.starting_time, x.ending_time, x.description, x.description as location, x.name, x.short_name, x.add_info_url, x.contact_name, x.contact_info FROM calitem x, calendar x, calitemtypes x WHERE x.calendar_id = x.calendar_id AND x.short_name LIKE ? AND x.style_id = 0 AND x.starting_date > to_char(sysdate-1, 'J') AND x.item_type_id = x.item_type_id AND ROWNUM <= 3 ORDER BY to_date(to_char(x.starting_date), 'J')
Adding the rownum attribute worked perfectly and the query was ordered before return. Thanks for the help
You should limit the query to return only the two most current rows. This can by achieved by a LIMIT clause (exists in MySQL, not sure about other DBs) and an ORDER BY clause.
There is not need to add an index that would count the rows returned by the ResultSet.