Develop Reference

`Could not open connection` issue despite using c3p0

I've been using c3p0 for connection pooling in my Spring Boot Application for a few months now. Everything was fine until about 2 weeks ago when I started experiencing connection issues especially in the morning. Every morning when I try to log-in to my application, it would throw a Could not open connection error. I would then restart my application in order to remove the problem. I am unable to figure out the root cause of the problem.
Here's my hibernate.cfg.xml:
hibernate-configuration>
<session-factory>
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.connection.url">jdbc:mysql://localhost:3306/mydb?autoReconnect=true</property>
<property name="hibernate.connection.username">root</property>
<property name="hibernate.connection.password">abc123</property>
<property name="hibernate.dialect">config.CustomDialect</property>
<property name="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.initialPoolSize">5</property>
<property name="hibernate.c3p0.minPoolSize">5</property>
<property name="hibernate.c3p0.maxPoolSize">100</property>
<property name="hibernate.c3p0.checkoutTimeout">3000</property>
<property name="hibernate.c3p0.maxStatementsPerConnection">30</property>
<property name="hibernate.c3p0.unreturnedConnectionTimeout">3000</property>
<property name="hibernate.c3p0.debugUnreturnedConnectionStackTraces">true</property>
<property name="show_sql">true</property>
<property name="format_sql">true</property>
<property name="hbm2ddl.auto">update</property>
...
POJO mappings
</session-factory>
</hibernate-configuration>
Here's my HibernateUtil Class:
public class HibernateUtil {
private static final SessionFactory sessionFactory;
static {
try {
Configuration configuration = new Configuration().configure();
StandardServiceRegistryBuilder builder = new StandardServiceRegistryBuilder()
.applySettings(configuration.getProperties());
sessionFactory = configuration.buildSessionFactory(builder.build());
} catch (Exception ex) {
throw new ExceptionInInitializerError(ex);
}
}
public Session openSession() {
return sessionFactory.openSession();
}
}
I've added c3p0 debug configuration to my application to cull unreturned connections(in-case of a memory leak) and generate a stack-trace for it but nothing shows up in the logs.
Here's some of the logs from this morning :
https://pastebin.com/MGb4Miau
Can anyone here help me figure where the problem lies?
EDIT: CustomDialect Class:
public class CustomDialect extends MySQL5InnoDBDialect {
public String getTableTypeString() {
return " ENGINE=InnoDB DEFAULT CHARSET=utf8";
}
}

This problem appear because you ran out of database connections.
too slow queries that hold the connection.
you have a lot of connection demands that can't fit the max pool size 100, it is the expected cause since the problem appear after a day of running.
or you have source leak, because you have not close the connection after ether the transaction success or failed.
From C3P0 logs on debug try to see how many connections are requested.
Also
Ideally, YOU must not use unreturnedConnectionTimeout in production, so you have to debug the connection leaks and when you have no more leaks remove
both unreturnedConnectionTimeout and debugUnreturnedConnectionStackTraces config.
Edit
try these config:
<property key="hibernate.connection.characterEncoding">UTF-8</property>
<property key="hibernate.connection.useUnicode">true</property>
Because sometimes the encoding in Hibernate is different from the encoding in MySQL db.

I finally figured out a solution to the problem. I made the following changes to my hibernate.cfg.xml:
<property name="hibernate.c3p0.maxIdleTime">10800</property>
<property name="hibernate.c3p0.maxIdleTimeExcessConnections">600</property>
And it works! The maxIdleTime property removes connections that have been idle from the pool for more than the specified time period(in seconds) which ensures that my connections are refreshed from time-to-time and maxIdleTimeExcessConnections lets me cull connections from the pool in excess of minPoolSize that have been idle for more than the specified time period(in seconds). This way I ensure that I don't have too many connections in the pool and that they all are fresh.

Why am I receiving JDBC driver warning and ThreadLocal errors?

I am running my application on GlassFish, I use Spring Security and Hibernate.
When I run the application the following warning and errors will be shown on GlassFish console. How can I avoid them?
WARNING: The web application [] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#1087985b]) and a value of type [org.hibernate.internal.SessionImpl] (value [SessionImpl(PersistenceContext[entityKeys=[],collectionKeys=[]];ActionQueue[insertions=[] updates=[] deletions=[] collectionCreations=[] collectionRemovals=[] collectionUpdates=[] unresolvedInsertDependencies=UnresolvedEntityInsertActions[]])]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
SEVERE: The web application [] created a ThreadLocal with key of type [net.sf.json.AbstractJSON$1] (value [net.sf.json.AbstractJSON$1#362386d7]) and a value of type [java.util.HashSet] (value [[]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
SEVERE: The web application [] created a ThreadLocal with key of type [net.sf.json.AbstractJSON$1] (value [net.sf.json.AbstractJSON$1#362386d7]) and a value of type [java.util.HashSet] (value [[]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
hibernate.cfg.xml
<hibernate-configuration>
<session-factory>
<!-- Database connection settings -->
<property name="connection.driver_class">
com.mysql.jdbc.Driver
</property>
<property name="connection.url">
jdbc:mysql://localhost:3306/myproject
</property>
<property name="connection.username">root</property>
<property name="connection.password"></property>
<!-- JDBC connection pool (use the built-in) -->
<property name="connection.pool_size">12</property>
<!-- SQL dialect -->
<property name="dialect">
org.hibernate.dialect.MySQLDialect
</property>
<!-- Enable Hibernate's automatic session context management -->
<property name="current_session_context_class">thread</property>
<!-- Disable the second-level cache -->
<!-- <property name="cache.provider_class">
org.hibernate.cache.EhCacheProvider
</property>
<property name="hibernate.cache.use_query_cache">true</property>-->
<!-- Echo all executed SQL to stdout -->
<property name="show_sql">true</property>
HibernateUtil.java
public class HibernateUtil {
private static ServiceRegistry serviceRegistry;
private static final ThreadLocal<Session> threadLocal = new ThreadLocal();
private static SessionFactory sessionFactory;
private static SessionFactory configureSessionFactory() {
try {
Configuration configuration = new Configuration();
configuration.configure();
serviceRegistry = new ServiceRegistryBuilder().applySettings(configuration.getProperties()).buildServiceRegistry();
sessionFactory = configuration.buildSessionFactory(serviceRegistry);
return sessionFactory;
} catch (HibernateException e) {
System.out.append("** Exception in SessionFactory **");
e.printStackTrace();
}
return sessionFactory;
}
static {
try {
sessionFactory = configureSessionFactory();
} catch (Exception e) {
System.err.println("%%%% Error Creating SessionFactory %%%%");
e.printStackTrace();
}
}
private HibernateUtil() {
}
public static SessionFactory getSessionFactory() {
return sessionFactory;
}
public static Session getSession() throws HibernateException {
Session session = threadLocal.get();
if (session == null || !session.isOpen()) {
if (sessionFactory == null) {
rebuildSessionFactory();
}
session = (sessionFactory != null) ? sessionFactory.openSession() : null;
threadLocal.set(session);
}
return session;
}
public static void rebuildSessionFactory() {
try {
sessionFactory = configureSessionFactory();
} catch (Exception e) {
System.err.println("%%%% Error Creating SessionFactory %%%%");
e.printStackTrace();
}
}
public static void closeSession() throws HibernateException {
Session session = (Session) threadLocal.get();
threadLocal.set(null);
if (session != null) {
session.close();
}
}
}

These are error messages that can happen in the case of application redeployments while the server is kept running.
If it's a shutdown scenario or a development redeployment these messages can be safely ignored, they only become important if you need redeployments in production, which is rare. Most of the times even in production we want to stop the server process and fully restart it. This is some details on each of the messages meaning:
Message 1 - driver not unregistered when the application stopped:
WARNING: The web application [] registered the JDBC driver
[com.mysql.jdbc.Driver] but failed to unregister it when the web
application was stopped. To prevent a memory leak, the JDBC Driver has
been forcibly unregistered.
JDBC drivers are registered at startup in a singleton at the level of the JVM, and this is meant to be done by the server, by publishing the driver jar in a folder at the server level.
In this case the application seems to carry the driver itself, which is not the way drivers are meant to be deployed.
To fix this remove the driver from the application and register it at the level of the server instead. If multiple applications have the same driver this will also cause memory leaks - see this answer as well.
Message 2 - ThreadLocal not cleaned up:
SEVERE: The web application [] created a ThreadLocal with key of
type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#1087985b])
and a value of type [org.hibernate.internal.SessionImpl] but
failed to remove it when the web application was stopped. Threads are
going to be renewed over time to try and avoid a probable memory leak.
This means that one application spring thread stored a Hibernate session in the thread (each thread as a data store where things can be attached via ThreadLocal).
But the thread did not clean up the session when the application was restarted, so this variable stored at the thread can be visible AFTER the redeployment when the thread gets reused.
This might be surprising but worst the session was pointing to other objects, which themselves where pointing to classes, which where pointing to the old classloader before the redeploy.
This means a large portion of the object tree will NOT be garbage collected due to this leaked 'link' to the objects of the previous deployment. The result is a ClassLoader Memory Leak.
The message says that this scenario MIGHT happen due to uncleaned ThreadLocals, and that some preventive measures are going to be put in place (start killing threads and creating new ones instead of pooling, to get rid of the leaked thread locals).
As a summary, these messages can be safely ignored if you don't need redeployments in production and always restart the server.

To get rid of the JDBC driver warning, run the following on application shutdown:
String url = "your JDBC url";
Driver driver = DriverManager.getDriver(url);
DriverManager.deregisterDriver(driver);
If you are using Spring beans, you can put this in the destroy() method of a DisposableBean.
In a servlet environment, you can use a ServletContextListener:
public class CleanupListener implements ServletContextListener {
public void contextDestroyed(ServletContextEvent arg0) {
// enter cleanup code here
}
public void contextInitialized(ServletContextEvent arg0) { }
}
Set it up in web.xml:
<listener>
<listener-class>com.example.CleanupListener</listener-class>
</listener>

JDK6 onward the JDBC driver automatically get loaded if any Driver class found in classpath without Class.forName() method and It may lead the such error messages. Better to write a Listener for your application and de-register every driver at the time of application shutdown. You can get all registered driver using DiverManager#getDrivers() method and can de-register one by one.

Hibernate doesn't want to load Oracle driver

I downloaded Hibernate 4.1.2 and am using Oracle Database 10g Release 2. The JDBC driver I am using is ojdbc14.jar.
I set up HibernateUtil class as:
public class HibernateUtil {
private static final SessionFactory sessionFactory = buildSessionFactory();
private static SessionFactory buildSessionFactory() {
// Create the SessionFactory from hibernate.cfg.xml
try{
Configuration configuration = new Configuration();
configuration.configure();
ServiceRegistry serviceRegistry = new ServiceRegistryBuilder().applySettings(configuration.getProperties()).buildServiceRegistry();
return configuration.buildSessionFactory(serviceRegistry);
}catch(HibernateException ex){
ex.printStackTrace();
throw ex;
}
}
public static SessionFactory getSessionFactory() {
return sessionFactory;
}
}
In hibernate.properties I have:
hibernate.dialect org.hibernate.dialect.OracleDialect
hibernate.connection.driver_class oracle.jdbc.driver.OracleDriver
hibernate.connection.username HR
hibernate.connection.password HR
hibernate.connection.url jdbc:oracle:thin:#localhost:1521/xe
But Hibernate doesn't want to load the driver. It throws an exception saying 'No appropriate driver found'.
I tried to load the driver with Class.forName("oracle.jdbc.driver.OracleDriver"); and it works fine.

The problem was in using the wrong JDBC Oracle driver. When I tried with ojdbc6.jar everything worked fine.

A couple of things:
Try to make the properties file valid by putting = between key and value
Check that there aren't any trailing spaces after the values
Use oracle.jdbc.OracleDriver instead of oracle.jdbc.driver.OracleDriver. See Difference between Oracle jdbc driver classes? for further reference.

Your connection URL is configured wrongly, should be:
hibernate.connection.url jdbc:oracle:thin:#localhost:1521:xe
More information for Oracle's URL can refer here.
As other answer point out:
Use oracle.jdbc.OracleDriver instead of oracle.jdbc.driver.OracleDriver

hibernate configuration for testing -(newbie wows)

I have been learning to use hibernate for a couple of months.
I am finding it difficult in deciding how to configure hibernate to work on a test database.
I have a hibernate.cfg.xml with db parameters given as elements.
<property name="connection.url">
jdbc:postgresql://localhost/mydb
</property>
<property name="connection.username">me</property>
<property name="connection.password">mypasswd</property>
My web app uses a HibernateUtil class which loads the configuration as below
class HibernateUtil {
private Class<T> persistentClass;
private static SessionFactory sessionFactory;
static {
try {
sessionFactory = new Configuration().configure().buildSessionFactory();
}catch (Throwable ex) {
throw new ExceptionInInitializerError(ex);
}
}
...
My dao implementation uses the above class to get the Session
public class BaseDaoImpl<T, ID extends Serializable>{
private Session session;
...
public Session getSession() {
if (session == null || !session.isOpen()){
return HibernateUtil.getCurrentSession();
}else{
return session;
}
}
public T findById(ID id){
T entity = (T) getSession().load(getPersistentClass(), id);
return entity;
}
This is OK as long as I work on the mydb configured in cfg.xml file.But for my tests I am using another database which is given in a test.properties file
test.db.url=jdbc:postgresql://localhost/mytestdb
test.db.driver=org.postgresql.Driver
test.db.username=testme
test.db.password=testpass
The only way I can make hibernate work on mytestdb is to manually replace every db related property in cfg.xml.I would very much like to use test_hibernate.cfg.xml with the test db properties,but since the configuration is done in a static block in HibernateUtil ,that won't work.
Is there a better way of configuring hibernate for these two databases?
I am using ant as build tool..
Any suggestions would be most welcome
jim

I would very much like to use test_hibernate.cfg.xml with the test db properties,but since the configuration is done in a static block in HibernateUtil
So then don't create your configuration in a static block.
Instead, create a class which accepts a parameter for the path to the configuration file (a la Spring's LocalSessionFactoryBean) which returns the Configuration/SessionFactory to use. Or if you truly want to stick with HibernateUtil (which is a strategy very much recommended against in any production setting), change it to read a property or system environment variable to read the configuration.

How can I prevent Hibernate + c3p0 + MySql creating large numbers of sleeping connections?

I'm using GWT with Hibernate, c3p0 and MySQL to produce a web app with a limited audience (max 50 users per day). During testing I found that Hibernate was opening a connection with each session but not closing it, irrespective of use of the close() method.
My current configuration is as follows:
hibernate.connection.driver_class=com.mysql.jdbc.Driver
hibernate.connection.url=
hibernate.connection.username=
hibernate.connection.password=
hibernate.dialect=org.hibernate.dialect.MySQLDialect
hibernate.current_session_context_class=thread
hibernate.c3p0.min_size=1
hibernate.c3p0.max_size=1
hibernate.c3p0.timeout=10
hibernate.c3p0.max_statements=50
hibernate.c3p0.idle_test_period=10
hibernate.c3p0.unreturned_connection_timeout=1
hibernate.connection.provider_class=org.hibernate.connection.C3P0ConnectionProvider
With each new connection to the application a new pool is created. For example if I set the pool size to 3, 2 connections to the application result in 6 connections until the application is closed.
The intended behaviour is to simply close or reuse the connections after each transaction. How can I achieve this?

During testing I found that Hibernate was opening a connection with each session but not closing it, irrespective of use of the close() method
When using a connection pool, calling Connection#close() doesn't physically close the connection but return it to the pool for future reuse. In other words, the connection stays open and that's the whole point of using a pool.
I call the following: AnnotationConfiguration().buildSessionFactory().getCurrentSession();
Well, that's the problem. You are creating a SessionFactory over and over (each creating its own pool) while you should create it only once for the lifetime of your application. If you are not using any particular framework, this is typically done in some utility class (the famous HibernateUtil class).
The official Hibernate Tutorial has a very basic example of such a class. Or see this one which is a bit richer.

The concept of a connection pool is exactly that. You have a pool of opened connnections, and when you need to do a transaction, you get a connection already opened. This way, you save a lot of time opening and closing connections. But you pay the price to keep the connections opened when you are not using them.
You have more info about c3p0 configuration
Update Apparently the OP was calling buildSessionFactory once per session. This has to be called once per lifetime of the application.
Here's the utility class that builds the sessionFactory of Hibernate and provides the session class to whoever asks for it. This is the cumberstone for a DAO class.
import org.hibernate.SessionFactory;
import org.hibernate.cfg.AnnotationConfiguration;
import org.hibernate.classic.Session;
public class HibernateUtil {
private static final SessionFactory sessionFactory;
static {
try {
// Create the SessionFactory from hibernate.cfg.xml
sessionFactory = new AnnotationConfiguration().configure().buildSessionFactory();
} catch (Throwable ex) {
// Make sure you log the exception, as it might be swallowed
System.err.println("Initial SessionFactory creation failed." + ex);
throw new ExceptionInInitializerError(ex);
}
}
public static SessionFactory getSessionFactory() {
return sessionFactory;
}
public static Session getCurrentSession() {
return sessionFactory.getCurrentSession();
}
}

Its not a good ideia to use a Connection Pool if you want to close the connection after each transaction. It's exacly what connection pools want to avoid... You should just turn off C3PO. Hibernate will handle the connection by himself (open and close as simple JDBC connection in each transaction)