Develop Reference

Android Java Efficient SQLite query with interruption

I have a table with > 200,000 rows (and might be a few million).
Sometimes the queries can take a few minutes to return a result.
I need to offer the end user the ability to abort the query and refine it.
To do that, I need to implement a query that can be cancelled or interrupted.
Is there a best practice for this?
The solution I came up with is to perform a series of smaller queries using LIMIT and OFFSET, checking for an interrupt between.
My concern is that repeatedly using LIMIT and OFFSET will result in searching the same data rows over and over, yielding a much slower query. Or does SQLite recognize it can start the next query at the row number where a prior identical search stopped?
public class InterruptableQuery {
public interface QueryResult {
void queryResult(boolean interrupted, List<MyData> results);
}
private QueryResult queryCallBack;
private volatile boolean _interrupted = false;
private volatile boolean _cancelled = false;
private SQLiteDatabase db;
public InterruptableQuery(SQLiteDatabase db, QueryResult qr) {
this.db = db;
queryCallBack = qr;
}
public void interruptQuery() { _interrupted = true; }
public void cancelQuery() { _cancelled = true; }
public void search(final String query) {
Thread searchThread = new Thread() {
#Override
public void run() {
boolean done = false;
int OFFSET = 100;
int offset = 0;
List<MyData> resultsList = new ArrayList<>();
while (!done && !_cancelled && !_interrupted) {
Cursor cursor = db.rawQuery(query + " LIMIT " + OFFSET + " OFFSET " + offset, null);
done = (cursor == null) || (cursor.getCount() < OFFSET);
if (cursor != null) {
if (cursor.moveToFirst()) {
do {
resultsList.add(new MyData(cursor));
} while (cursor.moveToNext());
}
cursor.close();
}
offset += OFFSET;
}
if (!_cancelled)
queryCallBack.queryResult(_interrupted, resultsList);
}
};
searchThread.start();
}
}

Simple query cursor in a thread is not finding value

For some reason in the following function the Cursor is not reading any marks. I don't know what I am doing wrong and I have been debugging this code for ours. When it runs it says the value of tagid and catid are both -1. No ping pong or pang :(
public String getCategoryNameByLawId(final int lawID){
final String[] categoryName = {"Success"+lawID};
final int[] tagID = {-1};
final int[] categoryID = {-1};
runnable = new Runnable() {
#Override
public void run() {
try {
openToRead();
String Query = "SELECT * from " + Constants.TABLE_LAW_TAG;
Cursor c1 = mSqLiteDatabase.rawQuery(Query, null);
if (c1.moveToFirst()) {
categoryName[0] = "ping";
while (c1.isAfterLast() == false) {
categoryName[0] = "pong";
try {
if (c1.getInt(c1.getColumnIndex(Constants.KEY_LAW_ID)) == lawID) {
int indexTagID = c1.getColumnIndex(Constants.KEY_TAG_ID);
tagID[0] = c1.getInt(indexTagID);
categoryName[0] = "pang";
}
} catch (Exception e) {
categoryName[0] = e.getMessage();
}
c1.moveToNext();
}
}
close();
openToRead();
Query = "SELECT * from " + Constants.TABLE_CATEGORY_TAG;
Cursor c2 = mSqLiteDatabase.rawQuery(Query, null);
if (c2.moveToFirst()) {
while (c2.isAfterLast() == false) {
if (c2.getInt(c2.getColumnIndex(Constants.KEY_TAG_ID)) == tagID[0]) {
int indexCategoryID = c2.getColumnIndex(Constants.KEY_CATEGORY_ID);
categoryID[0] = c2.getInt(indexCategoryID);
}
c2.moveToNext();
}
}
/*
exceptionHandler.alert(new RuntimeException(), "catid-" + categoryID[0]);
Query = "SELECT * from " + Constants.TABLE_CATEGORY;
Cursor c3 = mSqLiteDatabase.rawQuery(Query, null);
if (c3.moveToFirst()) {
while (c3.isAfterLast() == false) {
if (c3.getInt(c3.getColumnIndex(Constants.KEY_CATEGORY_ID)) == categoryID[0]) {
int indexCategoryName = c3.getColumnIndex(Constants.KEY_CATEGORY_NAME);
categoryName[0] = c3.getString(indexCategoryName);
}
c3.moveToNext();
}
}
exceptionHandler.alert(new RuntimeException(), "catnam-" + categoryName[0]);*/
close();
}
catch(Exception e){
categoryName[0] ="error";
}
}
};
new Thread(runnable).start();
return categoryName[0].toLowerCase() + " tagid: "+ tagID[0]+ " catid: "+ categoryID[0];
}

start() just starts the thread; it does not wait for it to finish.
The problem is that the value has not yet been found when you're executing return categoryName[0]....
Drop the Runnable indirection, and execute the database code directly.
If you really want to use a thread, you could wait for the thread to finish (call its join()), but this would not make sense because your main thread would be suspended as long as the database code is running. To be able to do other stuff in the main thread, you would have to reorganize your program so that the database code sends a separate message back to the main thread when it has the result. Have it look into CursorLoader (which needs a content provider, though).

Ormlite query takes time in Android

I am working in Android application in which I am using ormlite. I am taking my phone book contacts and saving them in my local database, but the problem is that it is taking too much time like for almost 1500 contact it is taking almost 70 seconds.
I searched for the Bulk insert in ormlite, but I can't figure it out how to implement it in my following code.
public static void loadLocalPhoneBookSample(Context ctx) {
try{
ContentResolver contentRes = ctx.getContentResolver();
Cursor cur = null;
String selection = ContactsContract.Contacts.HAS_PHONE_NUMBER;
cur = contentRes.query(ContactsContract.Contacts.CONTENT_URI, PROJECTIONS, selection, null, Phone.DISPLAY_NAME + " ASC");
context = ctx;
if (cur.getCount() > 0) {
// create DB object
MUrgencyDBHelper db = new MUrgencyDBHelper(ctx);
RuntimeExceptionDao<ContactLocal, ?> contactDAO = db.getContactLocalIntDataDao();
UpdateBuilder<ContactLocal, ?> updateDAO = contactDAO.updateBuilder();
try {
updateDAO.updateColumnValue("isUseless", true);
updateDAO.update();
} catch (SQLException e) {
e.printStackTrace();
}finally {
// db.writeUnlock();
}
while (cur.moveToNext()) {
String id = cur.getString(cur.getColumnIndex(ContactsContract.Contacts._ID));
/** read names **/
String displayName = cur.getString(cur.getColumnIndex(ContactsContract.Contacts.DISPLAY_NAME));
/** Phone Numbers **/
Cursor pCur = contentRes.query(ContactsContract.CommonDataKinds.Phone.CONTENT_URI,null, ContactsContract.CommonDataKinds.Phone.CONTACT_ID
+ " = ?", new String[] { id }, null);
while (pCur.moveToNext()) {
String number = pCur
.getString(pCur
.getColumnIndex(ContactsContract.CommonDataKinds.Phone.NUMBER));
String formatedNo = number.replaceAll("\\s+", "").replace("+", "00").replace("-", "").trim();
try {
QueryBuilder<ContactLocal, ?> query = contactDAO.queryBuilder();
query.where().eq("mFormatedNumber", number);
ContactLocal contact = query.queryForFirst();
boolean addContact = false, alreadyUpdated = true;
if (contact == null) {
addContact = true;
contact = new ContactLocal();
contact.setFirstName(displayName.trim());
contact.setLastName(displayName.trim());
contact.setContactNumber(formatedNo);
}
// check if this contact was already updated before
if (contact.getContactNumber() == null || contact.getContactNumber().length() == 0) {
contact.setContFirstLastNo(number, displayName, displayName, number);
alreadyUpdated = false;
}
contact.setUseless(false);
// if not updated already, Create/Update
if (addContact) {
contactDAO.create(contact);
} else
contactDAO.update(contact);
}
}
pCur.close();
}
}
}

the problem is that it is taking too much time like for almost 1500 contact it is taking almost 70 seconds
#CarloB has the right answer in terms of doing the mass creates inside the dao. callBatchTasks(...) method. Here's the docs on that subject:
http://ormlite.com/docs/batch
To make things a bit faster, you could also go through and record all of the mFormatedNumber in another List and then query for them using an IN query. Use a raw in query to get back the mFormatedNumber that are already in the database:
results = dao.queryRaw(
"SELECT mFormatedNumber from Contact WHERE mFormatedNumber IN ?",
mFormatedNumberList);
For using raw queries with ORMLite, see:
http://ormlite.com/docs/raw-queries
So then you would make one query to see which of the contacts need to be created and then do all of the inserts from within a batch transaction.
Otherwise you are doing ~3000 synchronous database transactions and 40/sec on an Android device is unfortunately pretty typical.

Here is my revised version (might need a few syntax changes)
public static void loadLocalPhoneBookSample(Context ctx) {
try {
ContentResolver contentRes = ctx.getContentResolver();
String selection = ContactsContract.Contacts.HAS_PHONE_NUMBER;
Cursor cur = contentRes.query(ContactsContract.Contacts.CONTENT_URI, PROJECTIONS, selection, null, Phone.DISPLAY_NAME + " ASC");
context = ctx;
if (cur.getCount() > 0) {
// create DB object
MUrgencyDBHelper db = new MUrgencyDBHelper(ctx);
RuntimeExceptionDao<ContactLocal, ?> contactDAO = db.getContactLocalIntDataDao();
UpdateBuilder<ContactLocal, ?> updateDAO = contactDAO.updateBuilder();
try {
updateDAO.updateColumnValue("isUseless", true);
updateDAO.update();
} catch (SQLException e) {
e.printStackTrace();
}finally {
// db.writeUnlock();
}
ArrayList<ContactLocal> contacts = new ArrayList<>();
while (cur.moveToNext()) {
String id = cur.getString(cur.getColumnIndex(ContactsContract.Contacts._ID));
/** read names **/
String displayName = cur.getString(cur.getColumnIndex(ContactsContract.Contacts.DISPLAY_NAME));
/** Phone Numbers **/
Cursor pCur = contentRes.query(ContactsContract.CommonDataKinds.Phone.CONTENT_URI,null, ContactsContract.CommonDataKinds.Phone.CONTACT_ID + " = ?", new String[] { id }, null);
while (pCur.moveToNext()) {
String number = pCur.getString(pCur.getColumnIndex(ContactsContract.CommonDataKinds.Phone.NUMBER));
String formatedNo = number.replaceAll("\\s+", "").replace("+", "00").replace("-", "").trim();
try {
QueryBuilder<ContactLocal, ?> query = contactDAO.queryBuilder();
query.where().eq("mFormatedNumber", number);
ContactLocal contact = query.queryForFirst();
if (contact == null) {
contact = new ContactLocal();
contact.setFirstName(displayName.trim());
contact.setLastName(displayName.trim());
contact.setContactNumber(formatedNo);
}
contact.setUseless(false);
contacts.add(contact);
}
}
pCur.close();
}
contactDao.callBatchTasks(new Callable<Void>() {
public Void call() throws Exception {
for (ContactLocal contact : contacts) {
contactDAO.createOrUpdate(contact);
}
}
});
}
}
The main optimization is to use callBatchTasks. From the ormlite documentation:
Databases by default commit changes after every SQL operation. This method disables this "auto-commit" behavior so a number of changes can be made faster and then committed all at once.
By creating an ArrayList and keeping track of the changes, you can use callBatchTasks to create/update at the end all in one shot.
Also I noticed that alreadyUpdated was never accessed, so it's safe to remove.
Also Dao has a createOrUpdate method which is the same as the addContact if statement you had before.

Android Multidimensional array

I have searched found a few answers but I am not quite sure I understand them. I want a multidimensional array or the equivalent of say string[0][1][1].
Here is what I have:
public List<List<List<String>>> loadCompleteExercises(String workout)
{
List<List<List<String>>> listExercises = new ArrayList<List<List<String>>>();
List<String> complete_time = new ArrayList<String>();
List<String> rest_time = new ArrayList<String>();
db = dbHelper.getReadableDatabase();
Cursor c = db.rawQuery("SELECT complete_time, rest_time FROM tbl_exercises WHERE workout = '"+workout+"';", null);
c.moveToFirst();
while(!c.isAfterLast()) {
try {
if(c.isNull(c.getColumnIndex("exercise"))) {
complete_time.add("00:00:05");
rest_time.add("00:00:00");
}else {
complete_time.add(c.getString(c.getColumnIndex("complete_time")));
rest_time.add(c.getString(c.getColumnIndex("rest_time")));
}
}catch (NullPointerException e)
{
Log.d("GET EXERCISES ERROR: ", e.toString());
}
c.moveToNext();
}
//listExercises.add();
return listExercises;
}
--- I want to add complete_time and rest_time to listExercises so that I can say do the following
listExercises.get(i).get(j) to yield the below
1 "00:00:05" "00:00:00"
2 "00:10:00" "00:10:00"
...
n "xx:xx:xx" "xx:xx:xx"

Try this, use a holder for both time, add them in a List
private class TimeHolder
{
public String completeTime;
public String restTime;
}
public List<TimeHolder> loadCompleteExercises(String workout)
{
List<TimeHolder> listExercises = new ArrayList<TimeHolder>();
db = dbHelper.getReadableDatabase();
Cursor c = db.rawQuery("SELECT complete_time, rest_time FROM tbl_exercises WHERE workout = '"+workout+"';", null);
c.moveToFirst();
while(!c.isAfterLast()) {
try
{
TimeHolder holder = new TimeHolder();
if(c.isNull(c.getColumnIndex("exercise"))) {
holder.completeTime = "00:00:05";
holder.restTime = "00:00:00";
}else {
holder.completeTime = c.getString(c.getColumnIndex("complete_time"));
holder.restTime = c.getString(c.getColumnIndex("rest_time"));
}
listExercises.add(holder);
}
catch (NullPointerException e)
{
Log.d("GET EXERCISES ERROR: ", e.toString());
}
c.moveToNext();
}
return listExercises;
}

data from sqlite into ArrayList<String>

i am using code this blog to have a draggable list. This tutorial is using a custom DragNDropAdapter
that takes the content as an ArrayList.
In my listActivity i query a table with returned column name.It has 11 values inserted.
i tried to convert it to ArrayList from String[] with many ways such as :
String[] from = new String[]{DbManager.KEY_NAME};
ArrayList<String> content = new ArrayList<String>();
for (int i=-1,l=from.length; ++i<l;) {
content.add(from[i]);
//Log.i("ArrayList", from[i]);
}
or
while(!mShopCatCursor.isAfterLast()){
content.add(mShopCatCursor.getString(0));
}
what i get is a list with just the name of the column, name.
do you have any ideas

You can use following method this method will get data from db and then return you an ArrayList of String for this data. In your case this array list will contain names.
private ArrayList<String> getArrayList() {
ArrayList<String> namesList = null;
Cursor cursor = null;
try {
String query = "";//your query here
cursor = db.rawQuery(query,null);
if (cursor != null && cursor.moveToFirst()) {
namesList = new ArrayList<String>();
do {
namesList.add(cursor.getString(0));
} while (cursor.moveToNext());
}
} catch (Exception e) {
e.printStackTrace();
namesList = null;
} finally {
if (cursor != null && !cursor.isClosed()) {
cursor.deactivate();
cursor.close();
cursor = null;
}
close();
}
return namesList;
}
/**
* Closes the database
*/
private void close() {
try {
if (db != null && db.isOpen()) {
DBHelper.close();
}
} catch (Exception e) {
e.printStackTrace();
}
}

String[] from = new String[]{DbManager.KEY_NAME};
Because your string array has only one value which is KEY_NAME.
What you need to do is,
Get values from Cursor using loop and populate it String[] above.
Cursor userCur = adaptor.getYourData();
if (userCur != null) {
String[] strArr = new String[userCur.getCount()];
startManagingCursor(userCur);
if (userCur.moveToFirst()) {
int count = 0;
do {
String userName = userCur.getString(1);
strArr[count] = userName.trim();
count++;
} while (userCur.moveToNext());
}
ArrayList<String> content = new ArrayList<String>();
for (int i=-1,l=from.length; ++i<l;) {
content.add(from[i]);
//Log.i("ArrayList", from[i]);
}
}
Note: I haven't validated this in IDE, there may be syntax errors.