Develop Reference

Confusion with DBCP connection pooling and Threads

I am trying to get Multi-Threading working in my Java web application and it seems like no matter what I try I run into some sort of issues with connection pooling.
My current process is that I have am looping through all my departments and processing them, which finally generates a display. This takes time, so I want to spawn a thread for each department and have them process concurrently.
After alot of time of first figuring out how to get my hibernate session to stay open in a thread to prevent the lazy initializion loading errors, I finally had the solution of creating a Spring bean of my Thread class, and creating a new instance of that bean for each thread. I have tried 2 different versions
1) I directly inject the DAO classes into the Bean. FAILED - After loading the page a few times I would get "Cannot get a connection, pool error Timeout waiting for idle object" for each thread and the app would crash.
2) Ok so then I tried injecting the spring SessionFactory into my bean, then create new instances of my DAO and set it with the SessionFactory. My DAO objects all extend HibernateDaoSupport. FAILED - After loading the page a few times I would get "too many connections" error messages.
Now what I am confused about is that my SessionFactory bean is a singleton which I understand to mean that it is a single Object that is shared throughout the Spring container. If that is true then why does it appear like each Thread is creating a new connection when it should just be sharing that single instance? It appears that all my connection pool is getting filled up and I don't understand why. Once the threads are done, all the connections that were created should be released but there not. I even tried running a close() operation on the injected SessionFactory but that has no effect. I tried limiting how many threads run concurrently at 5, hoping that would cause not so many connections to be created at one time but no luck.
I am obviously doing something wrong but I am not sure what. Am I taking the wrong approach entirely in trying to get my hibernate Session into my Threads? Am I somehow not managing my connection pool properly? any ideas would be greatly appreciated!
More info I thought of: My process creates about 25 threads total, which are run 5 at a time. I am able to refresh my page about 3 times before I start getting the errors. So apparently each refresh creates and holds on to a bunch of connections.
Here is my spring config file:
<bean id="mainDataSource" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
....
<!-- Purposely put big values in here trying to figure this issue out
since my standard smaller values didn't help. These big values Fail just the same -->
<property name="maxActive"><value>500</value></property>
<property name="maxIdle"><value>500</value></property>
<property name="minIdle"><value>500</value></property>
<property name="maxWait"><value>5000</value></property>
<property name="removeAbandoned"><value>true</value></property>
<property name="validationQuery"><value>select 0</value></property>
</bean>
<!--Failed attempt #1 -->
<bean id="threadBean" class="controller.manager.ManageApprovedFunds$TestThread" scope="prototype">
<property name="dao"><ref bean="dao" /></property>
</bean>
<!--Failed attempt #2 -->
<bean id="threadBean" class="manager.ManageApprovedFunds$TestThread" scope="prototype">
<property name="sessionFactory"><ref bean="sessionFactory" /></property>
</bean>
Java code:
ExecutorService taskExecutor = Executors.newFixedThreadPool(5);
for(Department dept : getDepartments()) {
TestThread t = (TestThread)springContext.getBean("threadBean");
t.init(dept, form, getSelectedYear());
taskExecutor.submit(t);
}
taskExecutor.shutdown();
public static class TestThread extends HibernateDaoSupport implements Runnable, ApplicationContextAware {
private ApplicationContext appContext;
#Override
public void setApplicationContext(ApplicationContext arg0)
throws BeansException {
this.appContext = arg0;
}
#Override
public void run() {
try {
MyDAO dao = new MyDAO();
dao.setSessionFactory(getSessionFactory());
//SOME READ OPERATIONS
getSessionFactory().close();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}

Basically you shouldn't try to share a single hibernate Session between threads. Hibernate sessions, nor the entity objects, are threadsafe and that can lead to some suprising problems. Here and [here]3[] is a small but nice read there is also some value in the comments. Basically don't try to share a Session between threads.
There is also another problem as the whole transaction management is based around ThreadLocal objects, the same goes for obtaining the current session and underlying JDBC connection. Now trying to spawn threads will lead to suprising problems, one of them would be connection pool starvation. (Note: don't open too many connections, more information here).
Next to not to opening to much connections you should be aware of starting to many threads. A Thread is bound to a cpu (or core if you have multiple cores). Adding to many threads might lead to heavy sharing of a single cpu/core between to many threads. This can kill your performance instead of increasing it.
In short IMHO your approach is wrong, each thread should simply read the entity it cares about, do its thing and commit the transaction. I would suggest using something like Spring Batch for this instead of inventing your own mechanism. (Although if it is simple enough I would probably go for it).

A database connection is not associated with the SessionFactory but with the Session. To get your handling correct you have to take care of the following:
Only create one instance of SessionFactory (per persistence context) - but you are doing this already
Don't close() the SessionFactory - it's lifetime should end when the application dies - that is at undeployment or server-shutdown.
Make sure to always close() your Session - you use one database connection per open Session and if these don't get closed you are leaking connections.

Catch startup exception in Grails if Hibernate database connection fails

We're building an app using Grails 2.0.4, GORM, and Hibernate. When the database is not available, Grails will not initialize, and startup fails. We thought our pool settings would protect against startup failures, but that doesn't seem to be the case.
If pool settings alone can't address this, is it possible to catch exceptions in resources.groovy where, if a database service can't be initialized, switch to a file-based service temporarily? Something like this...
resources.groovy
try{
myDataService(PostgresDatabaseServiceImpl){}
}catch(Exception e){
//if database connect failed, use local service instead
myDataService(FileBasedServiceImpl){}
}
Even if the above is possible, it creates a new problem; how to switch back, dynamically, once the database is available. We attempted the above try/catch, but it had no impact, the startup issue persists:
Error creating bean with name 'transactionManagerPostProcessor':
Initialization of bean failed
If it's possible to avoid startup failures through pool settings alone, we could certainly manage SQL exceptions at runtime when the app attempts to use bad database connections, but startup failures we can't manage.
DataSource.groovy (pool settings)
dataSource {
pooled = true
driverClassName = "org.postgresql.Driver"
properties {
maxActive = 20
minEvictableIdleTimeMillis=1800000
timeBetweenEvictionRunsMillis=1800000
numTestsPerEvictionRun=3
testOnBorrow=true
testWhileIdle=true
testOnReturn=true
validationQuery="SELECT 1"
}
}
hibernate {
cache.use_second_level_cache = false
cache.use_query_cache = false
cache.region.factory_class = 'net.sf.ehcache.hibernate.EhCacheRegionFactory'
}

We attempted the above try/catch, but it had no impact, the startup issue persists:
So it seems you already have the answer to the question of whether it's possible to register a Spring bean for a (potentially) unavailable database in resources.groovy.
As an alternative, you could try registering a Spring bean for the database at runtime. This advantage of this approach is that even if registering the bean fails, you will be able to catch the error and use the file-based service instead. An example of how to register DataSource beans at runtime is show here.
To use this approach, register only a bean for the file-based service in resources.groovy
myDataService(FileBasedServiceImpl)
Then when you need to access the datasource:
class DataSourceService implements ApplicationContextAware {
def myDataService
ApplicationContext applicationContext
private static PG_BEAN = 'postgres'
def getDataSource() {
try {
getPostgresService()
} catch (ex) {
myDataService
}
}
private getPostgresService() {
def postgres
if (applicationContext.containsBean(PG_BEAN)) {
postgres = applicationContext.getBean(PG_BEAN)
} else {
// register a bean under the name 'postGres' and store a reference to it in postgres
// https://stackoverflow.com/a/20634968/2648
}
checkPostgres(postgres)
}
private checkPostres(postgresBean) {
// check that the database is available, throw an exception if it's not, return
// postgresBean if it is
}
}

c3p0's max number of connections doesn't work in Linux

I have a java application that uses hibernate for O/R mapping and I also use c3p0 connection pooling that ships together with hibernate. The DB is Oracle.
This is in my hibernate.cfg.xml:
<property name="hibernate.c3p0.min_size">3</property>
<property name="hibernate.c3p0.max_size">5</property>
<property name="hibernate.c3p0.max_statements">20</property>
<property name="hibernate.c3p0.timeout">1800</property>
<property name="hibernate.c3p0.idleTestPeriod">120</property>
This is how I obtain the hibernate session:
public class HibernateUtil {
private static Logger log = Logger.getLogger(HibernateUtil.class);
private static final SessionFactory sessionFactory;
private static String CONFIG_FILE_LOCATION = "/hibernate.cfg.xml";
static {
try {
// Create the SessionFactory from hibernate.cfg.xml
sessionFactory = new Configuration().configure(CONFIG_FILE_LOCATION).buildSessionFactory();
} catch (Throwable ex) {
// Make sure you log the exception, as it might be swallowed
log.fatal("Initial SessionFactory creation failed." + ex.getMessage());
throw new ExceptionInInitializerError(ex);
}
}
public static SessionFactory getSessionFactory() {
return sessionFactory;
}
}
But when I run SELECT * FROM V$SESSION WHERE machine='xxx'; in Oracle the number of connections can reach 20, which is greater than max_size.
The max connections don't work in Linux environment but they do in Unix and Windows. Any setting in Linux that needs to be tweaked?
I'm also suspecting there is a cache of the application somewhere, as previously I set max_size of 20. I'm running the application in Tomcat 5.5. But I don't know of such an application cache in Tomcat.
Another info: I'm running Linux 2.6.9-34.ELsmp. Red Hat Enterprise Linux AS release 4 (Nahant Update 3)
Thanks.

I'd guess you have additional DB sessions opened not from your application. For example, there should be an additional connection - the one you're using to run the SELECT statement.

How can I prevent Hibernate + c3p0 + MySql creating large numbers of sleeping connections?

I'm using GWT with Hibernate, c3p0 and MySQL to produce a web app with a limited audience (max 50 users per day). During testing I found that Hibernate was opening a connection with each session but not closing it, irrespective of use of the close() method.
My current configuration is as follows:
hibernate.connection.driver_class=com.mysql.jdbc.Driver
hibernate.connection.url=
hibernate.connection.username=
hibernate.connection.password=
hibernate.dialect=org.hibernate.dialect.MySQLDialect
hibernate.current_session_context_class=thread
hibernate.c3p0.min_size=1
hibernate.c3p0.max_size=1
hibernate.c3p0.timeout=10
hibernate.c3p0.max_statements=50
hibernate.c3p0.idle_test_period=10
hibernate.c3p0.unreturned_connection_timeout=1
hibernate.connection.provider_class=org.hibernate.connection.C3P0ConnectionProvider
With each new connection to the application a new pool is created. For example if I set the pool size to 3, 2 connections to the application result in 6 connections until the application is closed.
The intended behaviour is to simply close or reuse the connections after each transaction. How can I achieve this?

During testing I found that Hibernate was opening a connection with each session but not closing it, irrespective of use of the close() method
When using a connection pool, calling Connection#close() doesn't physically close the connection but return it to the pool for future reuse. In other words, the connection stays open and that's the whole point of using a pool.
I call the following: AnnotationConfiguration().buildSessionFactory().getCurrentSession();
Well, that's the problem. You are creating a SessionFactory over and over (each creating its own pool) while you should create it only once for the lifetime of your application. If you are not using any particular framework, this is typically done in some utility class (the famous HibernateUtil class).
The official Hibernate Tutorial has a very basic example of such a class. Or see this one which is a bit richer.

The concept of a connection pool is exactly that. You have a pool of opened connnections, and when you need to do a transaction, you get a connection already opened. This way, you save a lot of time opening and closing connections. But you pay the price to keep the connections opened when you are not using them.
You have more info about c3p0 configuration
Update Apparently the OP was calling buildSessionFactory once per session. This has to be called once per lifetime of the application.
Here's the utility class that builds the sessionFactory of Hibernate and provides the session class to whoever asks for it. This is the cumberstone for a DAO class.
import org.hibernate.SessionFactory;
import org.hibernate.cfg.AnnotationConfiguration;
import org.hibernate.classic.Session;
public class HibernateUtil {
private static final SessionFactory sessionFactory;
static {
try {
// Create the SessionFactory from hibernate.cfg.xml
sessionFactory = new AnnotationConfiguration().configure().buildSessionFactory();
} catch (Throwable ex) {
// Make sure you log the exception, as it might be swallowed
System.err.println("Initial SessionFactory creation failed." + ex);
throw new ExceptionInInitializerError(ex);
}
}
public static SessionFactory getSessionFactory() {
return sessionFactory;
}
public static Session getCurrentSession() {
return sessionFactory.getCurrentSession();
}
}

Its not a good ideia to use a Connection Pool if you want to close the connection after each transaction. It's exacly what connection pools want to avoid... You should just turn off C3PO. Hibernate will handle the connection by himself (open and close as simple JDBC connection in each transaction)

Why does every thread in my application use a different hibernate session?

I have a web-application which uses hibernate and for some reason every thread (httprequest or other threads related to queueing) uses a different session.
I've implemented a HibernateSessionFactory class which looks like this:
public class HibernateSessionFactory {
private static final ThreadLocal<Session> threadLocal = new ThreadLocal<Session>();
private static Configuration configuration = new AnnotationConfiguration();
private static org.hibernate.SessionFactory sessionFactory;
static {
try {
configuration.configure(configFile);
sessionFactory = configuration.buildSessionFactory();
} catch (Exception e) {}
}
private HibernateSessionFactory() {}
public static Session getSession() throws HibernateException {
Session session = (Session) threadLocal.get();
if (session == null || !session.isOpen()) {
if (sessionFactory == null) {
rebuildSessionFactory();//This method basically does what the static init block does
}
session = (sessionFactory != null) ? sessionFactory.openSession(): null;
threadLocal.set(session);
}
return session;
}
//More non relevant methods here.
Now from my testing it seems that the threadLocal member is indeed initialized only once when the class is first loaded by the JVM but for some reason when different threads access the getSession() method they use different sessions. When a thread first accesses this class (Session) threadLocal.get(); will return null but as expected all other access requests will yeild the same session. I'm not sure how this can be happening as the threadLocal variable is final and the method threadLocal.set(session) is only used in the above context (which I'm 99.9% sure has to yeild a non null session as I would have encountered a NullPointerException at a different part of my app).
I'm not sure this is relevant but these are the main parts of my hibernate.cfg.xml file:
<hibernate-configuration>
<session-factory>
<property name="connection.url">someURL</property>
<property name="connection.driver_class"> com.microsoft.sqlserver.jdbc.SQLServerDriver</property>
<property name="dialect">org.hibernate.dialect.SQLServerDialect</property>
<property name="hibernate.connection.isolation">1</property>
<property name="hibernate.connection.username">User</property>
<property name="hibernate.connection.password">Password</property>
<property name="hibernate.connection.pool_size">10</property>
<property name="show_sql">false</property>
<property name="current_session_context_class">thread</property>
<property name="hibernate.hbm2ddl.auto">update</property>
<property name="hibernate.cache.use_second_level_cache">false</property>
<property name="hibernate.cache.provider_class">org.hibernate.cache.NoCacheProvider</property>
<!-- Mapping files -->
I'd appreciate any help granted and of course if anyone has any questions I'd be happy to clarify.
Ittai

Are you aware of the purpose of ThreadLocal?
From the docs:
This class provides thread-local
variables. These variables differ from
their normal counterparts in that each
thread that accesses one (via its get
or set method) has its own,
independently initialized copy of the
variable. ThreadLocal instances are
typically private static fields in
classes that wish to associate state
with a thread (e.g., a user ID or
Transaction ID).
You're getting a different Hibernate session for each thread precisely because that's what your code says to do.
Now, we can't really comment on whether this a good thing or not - although in many cases it's reasonably sensible. After all, you wouldn't want two threads to share the same session and interact with each other's transactions, would you?