Develop Reference

How can I efficiently refactor SQLite access in Java?

I am writing an Android app, in Java, which uses an SQLite database containing dozens of tables. I have a few Datasource classes set up to pull data from these tables and turn them into their respective objects. My problem is that I do not know the most efficient way to structure code that accesses the database in Java.
The Datasource classes are getting very repetitive and taking a long time to write. I would like to refactor the repetition into a parent class that will abstract away most of the work of accessing the database and creating objects.
The problem is, I am a PHP (loosely-typed) programmer and I'm having a very hard time solving this problem in a strictly-typed way.
Thinking in PHP, I'd do something like this:
public abstract class Datasource {
protected String table_name;
protected String entity_class_name;
public function get_all () {
// pseudo code -- assume db is a connection to our database, please.
Cursor cursor = db.query( "select * from {this.table_name}");
class_name = this.entity_class_name;
entity = new $class_name;
// loops through data in columns and populates the corresponding fields on each entity -- also dynamic
entity = this.populate_entity_with_db_hash( entity, cursor );
return entity;
}
}
public class ColonyDatasource extends Datasource {
public function ColonyDataSource( ) {
this.table_name = 'colony';
this.entity_class_name = 'Colony';
}
}
Then new ColonyDatasource.get_all() would get all the rows in table colony and return a bunch of Colony objects, and creating the data source for each table would be as easy as creating a class that has little more than a mapping of table information to class information.
Of course, the problem with this approach is that I have to declare my return types and can't use variable class names in Java. So now I'm stuck.
What should one do instead?
(I am aware that I could use a third-party ORM, but my question is how someone might solve this without one.)

First, is that you don't want to do these lines in your Java code:
class_name = this.entity_class_name;
entity = new $class_name;
It is possible to do what you are suggesting, and in languages such as Java it is called reflection.
https://en.wikipedia.org/wiki/Reflection_(computer_programming)
In this (and many cases) using reflection to do what you want is a bad idea for many reasons.
To list a few:
It is VERY expensive
You want the compiler to catch any mistakes, eliminating as many runtime errors as possible.
Java isn't really designed to be quacking like a duck: What's an example of duck typing in Java?
Your code should be structured in a different way to avoid this type of approach.
Sadly, I do believe that because it is strictly typed, you can't automate this part of your code:
// loops through data in columns and populates the corresponding fields on each entity -- also dynamic
entity = this.populate_entity_with_db_hash( entity, cursor );
Unless you do it through means of reflection. Or shift approaches entirely and begin serializing your objects (¡not recommending, just saying it's an option!). Or do something similar to Gson https://code.google.com/p/google-gson/. I.e. turn the db hash into a json representation and then using gson to turn that into an object.
What you could do, is automate the "get_all" portion of the object in the abstract class since that would be repetitive nearly every instance, but use an implementation so that you can have the abstract function rest assured that it can call a method of it's extending object. This will get you most of the way towards your "automated" approach, reducing the amount of code you must retype.
To do this we must consider the fact that Java has:
Generics (https://en.wikipedia.org/wiki/Generics_in_Java)
Function overloading.
Every Object in Java extends from the Object class, always.
Very Liskov-like https://en.wikipedia.org/wiki/Liskov_substitution_principle
Package scope: What is the default scope of a method in Java?
Try something like this (highly untested, and most likely wont compile) code:
// Notice default scoping
interface DataSourceInterface {
//This is to allow our GenericDataSource to call a method that isn't defined yet.
Object cursorToMe(Cursor cursor);
}
//Notice how we implement here?, but no implemented function declarations!
public abstract class GenericDataSource implements DataSourceInterface {
protected SQLiteDatabase database;
// and here we see Generics and Objects being friends to do what we want.
// This basically says ? (wildcard) will have a list of random things
// But we do know that these random things will extend from an Object
protected List<? extends Object> getAll(String table, String[] columns){
List<Object> items = new ArrayList<Object>();
Cursor cursor = database.query(table, columns, null, null, null, null,null);
cursor.moveToFirst();
while (!cursor.isAfterLast()) {
// And see how we can call "cursorToMe" without error!
// depending on the extending class, cursorToMe will return
// all sorts of different objects, but it will be an Object nonetheless!
Object object = this.cursorToMe(cursor);
items.add(object);
cursor.moveToNext();
}
// Make sure to close the cursor
cursor.close();
return items;
}
}
//Here we extend the abstract, which also has the implements.
// Therefore we must implement the function "cursorToMe"
public class ColonyDataSource extends GenericDataSource {
protected String[] allColumns = {
ColonyOpenHelper.COLONY_COLUMN_ID,
ColonyOpenHelper.COLONY_COLUMN_TITLE,
ColonyOpenHelper.COLONY_COLUMN_URL
};
// Notice our function overloading!
// This getAll is also changing the access modifier to allow more access
public List<Colony> getAll(){
//See how we are casting to the proper list type?
// Since we know that our getAll from super will return a list of Colonies.
return (List<Colony>)super.getAll(ColonyOpenHelper.COLONY_TABLE_NAME, allColumns);
}
//Notice, here we actually implement our db hash to object
// This is the part that would only be able to be done through reflection or what/not
// So it is better to just have your DataSource object do what it knows how to do.
public Colony cursorToMe(Cursor cursor) {
Colony colony = new Colony();
colony.setId(cursor.getLong(0));
colony.setTitle(cursor.getString(1));
colony.setUrl(cursor.getString(2));
return colony;
}
}

If your queries are virtually identical except for certain parameters, consider using prepared statements and binding
In SQLite, do prepared statements really improve performance?

So another option that I have yet to explore fully is something called Java Persistence API, there are projects that implement annotations very similar to this. The majority of these are in the form of an ORM which provide you with Data access objects (http://en.wikipedia.org/wiki/Data_access_object)
An open source project called "Hibernate" seems to be one of the go-to solutions for ORM in Java, but I have also heard that it is a very heavy solution. Especially for when you start considering a mobile app.
An android specific ORM solution is called OrmLite (http://ormlite.com/sqlite_java_android_orm.shtml), this is based off of Hibernate, but is very much stripped down and without as many dependencies for the very purpose of putting it on an android phone.
I have read that people using one will transition to the other very nicely.

Database information -> object: how should it be done?

My application will upon request retrieve information from a database and produce an object from that information. I'm currently considering two different techniques (but I'm open to others as well!) to complete this task:
Method one:
class Book {
private int id;
private String author;
private String title;
public Book(int id) {
ResultSet book = getBookFromDatabaseById(id);
this.id = book.id;
this.author = book.author;
// ...
}
}
Method two:
public class Book {
private HashMap<String, Object> propertyContainer;
public Book(int id) {
this.propertyContainer = getBookFromDatabaseById(id);
}
public Object getProperty(String propertyKey) {
return this.propertyContainer.get(propertyKey);
}
}
With method one, I believe that it's easier to control, limit and possibly access properties, adding new properties, however, becomes smoother with method two.
What's the proper way to do this?

I think this problem has been solved in many ways: ORM, DAO, row and table mapper, lots of others. There's no need to redo it again.
One issue you have to think hard about is coupling and cyclic dependencies between packages. You might think you're doing something clever by telling a model object how to persist itself, but one consequence of this design choice is coupling between model objects and the persistence tier. You can't use model objects without persistence if you do this. They really become one big, unwieldy package. There's no layering.
Another choice is to have model objects remain oblivious to whether or not they're persisted. It's a one way dependence that way: persistence knows about model objects, but not the other way around.
Google for those other solutions. There's no need to beat that dead horse again.

The first method will provide you with type safety for associated accessors so you will know what type of object you are getting back and don.t have to cast to that type the you are expecting (this becomes more important when providing anything other than primitives).
For that reason (plus that it will make the resulting code simpler and easier to read) I would pick the first one. In any large applications you will also be able to quickly, easily and neatly get parameter values back in the code for debug etc. within the object itself.
If anyone else is going to be working on this code also (or your planning on working it after you forget about it) the first one will also help as you know the parameters etc. The second one will only give you this with extensive javadoc.

The first one is the classical way. The second one is really tricky for nothing.

Giving a class member a reference to another classes members

On a scale of one to ten, how bad is the following from a perspective of safe programming practices? And if you find it worse than a five, what would you do instead?
My goal below is to get the data in the List of Maps in B into A. In this case, to me, it is ok if it is either a copy of the data or a reference to the original data. I found the approach below fastest, but I have a queasy feeling about it.
public class A {
private List<Map<String, String>> _list = null;
public A(B b) {
_list = b.getList();
}
}
public class B {
private List<Map<String, String>> _list = new ArrayList<Map<String, String>>();
public List<Map<String, String>> getList() {
// Put some data in _list just for the sake of this example...
_list.add(new HashMap<String, String>());
return _list;
}
}

The underlying problem is a bit more complex:
From a security perspective, this is very, very bad.
From a performance perspective, this is very, very good.
From a testing perspective, it's good because there is nothing in the class that you can't easily reach from a test
From an encapsulation perspective, it's bad since you expose the inner state of your class.
From a coding safety perspective, it's bad because someone will eventually abuse this for some "neat" trick that will cause odd errors elsewhere and you will waste a lot of time to debug this.
From an API perspective, it can be either: It's hard to imagine an API to be more simple but at the same time, it doesn't communicate your intent and things will break badly if you ever need to change the underlying data structure.
When designing software, you need to keep all of these points in the back of your mind. With time, you will get a feeling which kinds of errors you make and how to avoid them. Computers being as dump and slow as they are, there is never a perfect solution. You can just strive to make it as good as you can make it at the when you write it.
If you want to code defensively, you should always copy any data that you get or expose. Of course, if "data" is your whole data model, then you simply can't copy everything each time you call a method.
Solutions to this deadlock:
Use immutables as often as you can. Immutables and value objects are created and never change after that. These are always safe and the performance is OK unless the creation is very expensive. Lazy creation would help here but that is usually its own can of worms. Guava offers a comprehensive set of collections which can't be changed after creation.
Don't rely too much on Collections.unmodifiable* because the backing collection can still change.
Use copy-on-write data structures. The problem above would go away if the underlying list would clone itself as soon as A or B start to change it. That would give each its own copy effectively isolation them from each other. Unfortunately, Java doesn't have support for these built in.

In this case, to me, it is ok if it is either a copy of the data or a reference to the original data.
That is the sticking point.
Passing the object instance around is the fastest, but allows the caller to change it, and also makes later changes visible (there is no snapshot).
Usually, that is not a problem, since the caller is not malicious (but you may want to protect against coding errors).
If you do not want the caller to make changes, you could wrap it into an immutable wrapper.
If you need a snapshot, you can clone the list.
Either way, this will only snapshot/protect the list itself, not its individual elements. If those are mutable, the same reasoning applies again.

I would say that you will have too choose between efficiency and encapsulation. By directly accessing a member of the class it will have its state changed. That might be unexpected and lead to nasty surprises. I would also say that it increases the coupling between the two classes.
An alternative is to let the information expert principle decide and leave the job to the class that have the information. You will have to judge if the work that was suppose to be done with class A really is the responsibility of class B.
But really, speed and clean code can be conflicting interests. Some times you just have to play dirty to get it quick enough.

All you're creating is a reference to B._list. So 10 if you wanted to copy the items.
You could iterate over all b._list items and add them to the A._list manually:
public A(B b) {
_list = new List<Map<String, String>> ();
for (Map<String,String> map : b.getList()) {
Map<String,String> newMap = new HashMap<String,String>();
while(map.keySet().iterator().hasNext()) {
String key = map.keySet().iterator().next();
newMap.put(key,map.get(key));
}
_list.add(newMap);
}

Use a HashMap to store instance variables?

I would like to create a base class that all classes in my program will extend. One thing I wanted to do was find a uniform way to store all instance variables inside the object.
What I have come up with is to use a HashMap to store the key/value pairs for the object and then expose those values through a get and set method.
The code that I have for this so far is as follows:
package ocaff;
import java.util.HashMap;
public class OcaffObject {
private HashMap<String, Object> data;
public OcaffObject() {
this.data = new HashMap<String, Object>();
}
public Object get(String value) {
return this.data.get(value);
}
public void set(String key, Object value) {
this.data.put(key, value);
}
}
While functionally this works, I am curious if there are any real issues with this implementation or if there is a better way to do this?
In my day to day work I am a PHP programmer and my goal was to mimic functionality that I used in PHP in Java.

I don't think this is a good way to deal with what you mean.
Programming in java is quite different than programming in php, from my point of view.
You need to keep things clean and strongly typed, using the real paradigm of clean object oriented programming.
Some problems with this technique comes to my mind, here are some, not in importance order.
First problem you have with this is performance and memory footprint: this will consume a lot of memory and will perform very badly.
Second problem is concurrency, HashMap is not thread safe.
Third problem is type safety: you don't have type safety anymore, you can write to a field whatever you want and no one is checking it, a real anti-pattern.
Fourth problem is debugging... it will be hard to debug your code.
Fifth problem is: everyone can write and read any field knowing his name.
Sixth problem: when you change the name of a field in the hash set you don't get any kind of compile time error, you only get strange run-time behavior. Refactoring will become impossible.
Typed fields are much more useful and clean.

If you're taking the time to make a class for this, I would simply add what you need as members of the class. This will give you compile time checking of your class members, greatly reducing your subtle bugs.

Looking for a drop-in replacement for a java.util.Map

Problem
Following up on this question, it seems that a file- or disk-based Map implementation may be the right solution to the problems I mentioned there. Short version:
Right now, I have a Map implemented as a ConcurrentHashMap.
Entries are added to it continually, at a fairly fixed rate. Details on this later.
Eventually, no matter what, this means the JVM runs out of heap space.
At work, it was (strongly) suggested that I solve this problem using SQLite, but after asking that previous question, I don't think that a database is the right tool for this job. So - let me know if this sounds crazy - I think a better solution would be a Map stored on disk.
Bad idea: implement this myself. Better idea: use someone else's library! Which one?
Requirements
Must-haves:
Free.
Persistent. The data needs to stick around between JVM restarts.
Some sort of searchability. Yes, I need the ability to retrieve this darn data as well as put it away. Basic result set filtering is a plus.
Platform-independent. Needs to be production-deployable on Windows or Linux machines.
Purgeable. Disk space is finite, just like heap space. I need to get rid of entries that are n days old. It's not a big deal if I have to do this manually.
Nice-to-haves:
Easy to use. It would be great if I could get this working by the end of the week.
Better still: the end of the day. It would be really, really great if I could add one JAR to my classpath, change new ConcurrentHashMap<Foo, Bar>(); to new SomeDiskStoredMap<Foo, Bar>();
and be done.
Decent scalability and performance. Worst case: new entries are added (on average) 3 times per second, every second, all day long, every day. However, inserts won't always happen that smoothly. It might be (no inserts for an hour) then (insert 10,000 objects at once).
Possible Solutions
Ehcache? I've never used it before. It was a suggested solution to my previous question.
Berkeley DB? Again, I've never used it, and I really don't know anything about it.
Hadoop (and which subproject)? Haven't used it. Based on these docs, its cross-platform-readiness is ambiguous to me. I don't need distributed operation in the foreseeable future.
A SQLite JDBC driver after all?
???
Ehcache and Berkeley DB both look reasonable right now. Any particular recommendations in either direction?

UPDATE (some 4 years after first post...): beware that in newer versions of ehcache, persistence of cache items is available only in the pay product. Thanks #boday for pointing this out.
ehcache is great. It will give you the flexibility you need to implement the map in memory, disk or memory with spillover to disk. If you use this very simple wrapper for java.util.Map then using it is blindingly simple:
import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.Set;
import net.sf.ehcache.Cache;
import net.sf.ehcache.Element;
import org.apache.log4j.Logger;
import com.google.common.collect.Sets;
public class EhCacheMapAdapter<K,V> implements Map<K,V> {
#SuppressWarnings("unused")
private final static Logger logger = Logger
.getLogger(EhCacheMapAdapter.class);
public Cache ehCache;
public EhCacheMapAdapter(Cache ehCache) {
super();
this.ehCache = ehCache;
} // end constructor
#Override
public void clear() {
ehCache.removeAll();
} // end method
#Override
public boolean containsKey(Object key) {
return ehCache.isKeyInCache(key);
} // end method
#Override
public boolean containsValue(Object value) {
return ehCache.isValueInCache(value);
} // end method
#Override
public Set<Entry<K, V>> entrySet() {
throw new UnsupportedOperationException();
} // end method
#SuppressWarnings("unchecked")
#Override
public V get(Object key) {
if( key == null ) return null;
Element element = ehCache.get(key);
if( element == null ) return null;
return (V)element.getObjectValue();
} // end method
#Override
public boolean isEmpty() {
return ehCache.getSize() == 0;
} // end method
#SuppressWarnings("unchecked")
#Override
public Set<K> keySet() {
List<K> l = ehCache.getKeys();
return Sets.newHashSet(l);
} // end method
#SuppressWarnings("unchecked")
#Override
public V put(K key, V value) {
Object o = this.get(key);
if( o != null ) return (V)o;
Element e = new Element(key,value);
ehCache.put(e);
return null;
} // end method
#Override
public V remove(Object key) {
V retObj = null;
if( this.containsKey(key) ) {
retObj = this.get(key);
} // end if
ehCache.remove(key);
return retObj;
} // end method
#Override
public int size() {
return ehCache.getSize();
} // end method
#Override
public Collection<V> values() {
throw new UnsupportedOperationException();
} // end method
#Override
public void putAll(Map<? extends K, ? extends V> m) {
for( K key : m.keySet() ) {
this.put(key, m.get(key));
} // end for
} // end method
} // end class

Have you never heard about prevalence frameworks ?
EDIT some clarifications on the term.
Like James Gosling now says, no SQL DB is as efficient as an in-memory storage. Prevalence frameworks (most known being prevayler and space4j) are built upon this idea of an in-memory, maybe storable on disk, storage. How do they work ? In fact, it's deceptively simple : a storage object contains all persistent entities. This storage can only be changed by serializable operations. As a consequence, putting an object in storage is a Put operation performed in isolated context. As this operation is serializable, it may (depending upon configuration) be also saved on disk for long-term persistence. However, the main data repository is the memory, which proides undoubtly fast access time, at the cost of a high memory usage.
Another advantage is that, because of their obvious simplicity, these frameworks hardly contain more than a tenth of classes
Considering your question, the use of Space4J immediatly came to my mind (as it provides support for "passivation" of rarely used objects, that's to say their index key is in memory, but the objects are kept on disk as long as they're not used).
Notice you can also find some infos at c2wiki.

Berkeley DB Java Edition has a Collections API. Within that API, StoredMap in particular, is a drop-in replacement for a ConcurrentHashMap. You'll need to create the Environment and Database before creating the StoredMap, but the Collections tutorial should make that pretty easy.
Per your requirements, Berkeley DB is designed to be easy to use and I think that you'll find that it has exceptional scalability and performance. Berkeley DB is available under an open source license, it's persistent, platform independent and allows you to search for data. The data can certainly be purged/deleted, as needed. Berkeley DB has long list of other features which you may find highly useful to your application, especially as your requirements change and grow with the success of the application.
If you decide to use Berkeley DB Java Edition, please be sure to ask questions on the BDB JE Forum. There's an active developer community that's happy to help answer questions and resolve problems.

We have a similar solution implemented using Xapian. It's fast, it's scalable, it provedes almost all search functionality you requested, it's free, multiplatform, and of course purgeable.

I came accross jdbm2 a few weeks ago. The usage is very simple. You should be able to get it to work in half an hour. One drawback is that the object which is put into the map must be serializable, i.e. implement Serializable. Other Cons are given in their website.
However, all object persistence database are not a permanent solution for storing objects of you own java class. If you decide to make change to the fields of the class, you will no longer be able to reteive the object from the map collection. It is ideal to store standard serializable classes line String, Integer, etc.

The google-collections library, part of http://code.google.com/p/guava-libraries/, has some really useful Map tools. MapMaker in particular lets you make concurrent HashMaps with timed evictions, soft values that will be swept up by the garbage collector if you're running out of heap, and computing functions.
Map<String, String> cache = new MapMaker()
.softValues()
.expiration(30, TimeUnit.MINUTES)
.makeComputingMap(new Function<String, String>() {
#Override
public String apply(String input) {
// Work out what the value should be
return null;
}
});
That will give you a Map cache that will clean up after itself and can work out its values. If you're able to compute values like that then great, otherwise it would map perfectly onto http://redis.io/ which you'd be writing into (to be fair, redis would probably be fast enough on its own!).