Develop Reference

Used and Unused db connection objects

I encountered a question where in we have a set of say DB connections and multiple users simultaneously request for them.
I must be able to provide them the Conn object from a pool of existing connectios, and once they are done using it, put it back into the same pool of available connections.
The approach I could think of was to have 2 sets for each used and unused connections and keep moving objects back and forth.
I am somehow not convinced it's the best way to do this.
Can anyone pls suggest a nicer approach?
I was wondering if we could mark the connection obj as in use or something so that we can do away with these two sets?

TL;DR: Use a connection pool
There are thousands of implementations to choose from:
Hikari CP
Bone CP
C3P0
Commons DBCP
and others
These pools are tested, working, optimised code. Not whatever your homebrew solution is.
The problem you encountered is just one you will encounter on the way to writing a resilient connection pool. The solution you propose is naive at best. There are many other issues, from the misuse of the pool to connection dropout that you have to handle. And not only that but the code must be threadsafe.
For general scarce resource pooling, not JDBC connections, use Commons Pool. Again, this is tested, working code.
Do not reinvent the square wheel.

Don't reinvent the wheel. There are existing solutions for this, one of them being Apache Commons pools : https://commons.apache.org/proper/commons-pool/. All you have to do is provide a factory to initialize your connections at the beginning, then Apache will provide you with a GenericObjectPool (see doc) on which you can call borrowObject before accessing a connection in the pool and then release it using returnObject.

Using Spring JDBC Template - Performance

Is Spring SimpleJDBCTemplate production grade?
How good is it to use in the high volume (traffic) site? (Lets say 1K hits per second)
What are the choices we have to improve the performance (Connection Pooling, etc) on the Application Level?

Yes, it is production grade. Your limit at that volume isn't ran JDBC vs Spring JDBC template vs whatever. Your limit is resources if it is truly 1000 hits per second. 1000 users at the same time is a lot less hits because they they have "think time" rather than continuously hitting your app.
With any database technology, you will need to tune a lot, look at caching and be very aware of resources. And yes, you absolutely need connection pooling.

If you are talking of using SimpleJDBCTemplate over pure JDBC...go for it! It should definitely give as much performance and help you get rid of most JDBC connection make/break boiler plate code and exception handling etc.
Spring would allow you to configure connection pooling, tuning as you would do using pure JDBC.
Heres an old blog to provide you with performance comparison: http://pramatr.wordpress.com/2008/08/19/spring-jdbctemplate-the-phantom-performance-problem/
Spring has become better since.

How to connect Derby Database with Servlet?

I have never connected to a database in java before. May I know if I should go about accessing a derby database with servlet?
I have checked this: How do I access a database from my servlet or JSP?
But I saw comments on the article saying that this is a bad way to connect. Could any one explain or show me the best way to that I should code to access my derby database?
Thank you very much.

They are all right indeed, in suggesting that. We don't don't access database directly from Servlets or JSPs, these both are meant to be web tier, isn't it?
So, what to do? Grab a JDBC tutorial. The official one is an excellent choice here. That will give you a good idea about connecting to database from Java, and grasp over JDBC API. After that you should go and read about DAO pattern, and how we employ that in real apps.
Moreover, I think you also should read about MVC pattern, because it seems to me that you are not very clear on that as well.
Once you understand all these and come up with a toy like application using all these stuff. Next step would be to have a look into Connection Pooling mechanism.

Since you are using servelt you must be using a container line Apache Tomcat. You should look to define a connection pool like this http://tomcat.apache.org/tomcat-5.5-doc/jndi-datasource-examples-howto.html. If you are using any other container then that will also have similar setup.
Other option is to create a separate DBManager kind of class which looks after initializing and returning connection. This class you can use in the servlet.

Using JDBC and having your app server's application pool is a good start. You can also use some API to make your life easier like Hibernate.

It is a "bad way", because it doesn't make use of a (JNDI-managed) connection pool to obtain connections. Although acquiring a connection costs "only" a few hundred milliseconds, this has impact in a busy multiuser environment. A connection pool will worry about opening and closing connections and release them immediately on every getConnection() call so that it effectively costs almost zero milliseconds. If you sum that up in a busy multiuser environment, then the differences are noticeable.
A connection pool is usually to be configured in flavor of a JNDI datasource which is managed by the servletcontainer in question. As you didn't mention which one you're using, I can at highest point to one of my answers which contains a Tomcat 6.0 targeted example: here.
Hope this helps.

How can I configure hibernate to use context-specific connection information?

I'm writing a Java SE (Note, not Java EE) application using Hibernate, and I need to provide a different connection to Hibernate for each thread of execution. These connections must be pooled, and each one has at the very least different authentication and, possibly, a different JDBC URL. The connections will be re-used (as can be inferred from the pooling requirement).
What parts of Hibernate/C3P0/et al do I have to override? Can this be accomplished with those tools, or do I need to write my own pooling data source?

I think the best course of action would be creating a SessionFactory for each data source, with possibly pooled connections - that's what's eqbridges suggested in his answer.
Now, Hibernate does have a ConnectionProvider hook, so I suppose you could write an implementation that would return Connections to different data sources, depending on current thread of execution and some additional parameters. Theoretically, you can then have one SessionFactory instance, which will be using different connections to different databases, supplied by your custom ConnectionProvider implementation. But, one SessionFactory holds quite a bit of data, and that data is then used by Hibernate internally, when opening a Session for a unit of work. Plus, there's a second-level cache associated with it as well.
Unfortunately, how will the factory and Sessions you open from it behave in the face of such a provider is anybody's guess. It feels like a hack to me, and I doubt it was ever considered a viable use-case for a SessionFactory. It can possibly lead to all kinds of, possibly very subtle, bugs or data corruption.
On another note, be sure to exactly measure the cost of creating multiple SessionFactories - it may not be as high as you think. Be sure to compare that with the cost of simply opening the needed JDBC connections. I don't know what kind of results you might get, but I think you should be sure about performance before you resort to more hackish solutions.

You have two questions here:
Connections are not thread safe, so each thread must have its own connection. Since you're working with Hibernate, what your application sees is actually a Session obtained from a SessionFactory. To utilize this, you call the SessionFactory#getCurrentSession() method, and configure the current session context in hibernate.cfg.xml:
<property name="current\_session\_context\_class">thread</property>
If you've properly configured thread pooling (using c3po or whatever pooling mechanism you favor) in hibernate.cfg.xml, then each thread will get a connection from that pool.
To maintain multiple data sources that the application may need to work with, then you need to configure a separate SessionFactory for each JDBC url you'd like to access. In your application you'll need to have some means of selecting with SessionFactory you'll need to choose (e.g. "client ID"), using this you can manage each of the SessionFactory instances in a Map or somesuch data structure (in a Java EE app you'd get a reference from JNDI).
To summarize (and generalize), basically a SessionFactory is essentially huge wrapper around a DataSource (and attendant connection pool). It is read-only (and hence thread safe), heavyweight and static, constructed once, and knows everything it needs to about a given DataSource.
A Session, on the other hand is essentially a lightweight wrapper around a Connection. It is not thread safe, often short-lived, and intended to be used and then thrown away.
Hope this helps!

Connection pooling in hsqldb

What is the best way to implement connection pooling in hsqldb, without compromising on the speed?

Hibernate gets connections from a DataSource, uses them and closes them. You need a connection pool or it will be very inefficient, consuming a lot of resources both on your app and on the DBMS, regardless of the database server you use.
You should try out commons-dbcp from Apache-Jakarta, it's very efficient and really simple to set up. It depends on commons-pool.
You just define a BasicDataSource with DBCP and it will manage the connections from whatever JDBC driver you tell it to use. It has connection validation and lots of other stuff.
Of, if you're writing a web app, configure a connection pool on the container you will be using and use that, instead of defining your own pool.

You are comparing apples and oranges:
If you want orm compare the performance of different orm tools against the same db.
If you want connection pooling compare different connection pooling libraries against the same db.
Performing ORM incurs extra effort so it will never be as fast as direct JDBC access. That said, hibernate goes to great lengths (and very successfuly) to minimise this additional overhead. With ORM you are trading off significantly increased development productivity against a relatively small drop in performance.
Connection pooling is an orthogonal problem to orm. Most obviously, hibernate allows you to select your own connection pooling infrastructure.
Also, be aware that in practise there is often a fairly tight coupling between connection pooling and transaction mgmt. For example, a typical J2EE application will leave connection pooling to the container (via the JDBC Datasource API) and rely on declarative transactions. In this case connections and transactions are managed (approximately) together.
If you aren't in a J2EE container and you don't need orm I would simply compare C3P0, commons-pool, etc.

If you used Hibernate's own connection pooler, you could consider using c3p0? (If you're already using c3p0, I can't help further) I haven't used HSQLDB myself but I think that could be worth trying.