Develop Reference

`Could not open connection` issue despite using c3p0

I've been using c3p0 for connection pooling in my Spring Boot Application for a few months now. Everything was fine until about 2 weeks ago when I started experiencing connection issues especially in the morning. Every morning when I try to log-in to my application, it would throw a Could not open connection error. I would then restart my application in order to remove the problem. I am unable to figure out the root cause of the problem.
Here's my hibernate.cfg.xml:
hibernate-configuration>
<session-factory>
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.connection.url">jdbc:mysql://localhost:3306/mydb?autoReconnect=true</property>
<property name="hibernate.connection.username">root</property>
<property name="hibernate.connection.password">abc123</property>
<property name="hibernate.dialect">config.CustomDialect</property>
<property name="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.initialPoolSize">5</property>
<property name="hibernate.c3p0.minPoolSize">5</property>
<property name="hibernate.c3p0.maxPoolSize">100</property>
<property name="hibernate.c3p0.checkoutTimeout">3000</property>
<property name="hibernate.c3p0.maxStatementsPerConnection">30</property>
<property name="hibernate.c3p0.unreturnedConnectionTimeout">3000</property>
<property name="hibernate.c3p0.debugUnreturnedConnectionStackTraces">true</property>
<property name="show_sql">true</property>
<property name="format_sql">true</property>
<property name="hbm2ddl.auto">update</property>
...
POJO mappings
</session-factory>
</hibernate-configuration>
Here's my HibernateUtil Class:
public class HibernateUtil {
private static final SessionFactory sessionFactory;
static {
try {
Configuration configuration = new Configuration().configure();
StandardServiceRegistryBuilder builder = new StandardServiceRegistryBuilder()
.applySettings(configuration.getProperties());
sessionFactory = configuration.buildSessionFactory(builder.build());
} catch (Exception ex) {
throw new ExceptionInInitializerError(ex);
}
}
public Session openSession() {
return sessionFactory.openSession();
}
}
I've added c3p0 debug configuration to my application to cull unreturned connections(in-case of a memory leak) and generate a stack-trace for it but nothing shows up in the logs.
Here's some of the logs from this morning :
https://pastebin.com/MGb4Miau
Can anyone here help me figure where the problem lies?
EDIT: CustomDialect Class:
public class CustomDialect extends MySQL5InnoDBDialect {
public String getTableTypeString() {
return " ENGINE=InnoDB DEFAULT CHARSET=utf8";
}
}

This problem appear because you ran out of database connections.
too slow queries that hold the connection.
you have a lot of connection demands that can't fit the max pool size 100, it is the expected cause since the problem appear after a day of running.
or you have source leak, because you have not close the connection after ether the transaction success or failed.
From C3P0 logs on debug try to see how many connections are requested.
Also
Ideally, YOU must not use unreturnedConnectionTimeout in production, so you have to debug the connection leaks and when you have no more leaks remove
both unreturnedConnectionTimeout and debugUnreturnedConnectionStackTraces config.
Edit
try these config:
<property key="hibernate.connection.characterEncoding">UTF-8</property>
<property key="hibernate.connection.useUnicode">true</property>
Because sometimes the encoding in Hibernate is different from the encoding in MySQL db.

I finally figured out a solution to the problem. I made the following changes to my hibernate.cfg.xml:
<property name="hibernate.c3p0.maxIdleTime">10800</property>
<property name="hibernate.c3p0.maxIdleTimeExcessConnections">600</property>
And it works! The maxIdleTime property removes connections that have been idle from the pool for more than the specified time period(in seconds) which ensures that my connections are refreshed from time-to-time and maxIdleTimeExcessConnections lets me cull connections from the pool in excess of minPoolSize that have been idle for more than the specified time period(in seconds). This way I ensure that I don't have too many connections in the pool and that they all are fresh.

Why Hibernate doesn't save my object in database

I'm using Hibernate 4.3.10 with Java 8.
To be more explicit see the following code example :
public static long createBlindStructure(BlindStructure pBlindStructure){
Transaction tcx = null;
SessionFactory factory = HibernateUtil.getSessionFactory();
Session session = factory.openSession();
int id = -1;
try{
tcx = session.beginTransaction();
id = session.save(pBlindStructure);
tcx.commit();
}
catch( Throwable e){
tcx.rollback();
}
finally{
session.close();
}
return id;
}
In my mind this method save() open session and transaction, save my object and close session and transaction. From save() I tried to get back the identifier as describe in javadoc. But It doesn't work, I see the request execute in my log (thanks to Hibernate debug mode).
Hibernate: insert into PokerLeagueManager.blindStructure (structureJson) values (?)
But when I try this :
public static long createBlindStructure(BlindStructure pBlindStructure){
Transaction tcx = null;
SessionFactory factory = HibernateUtil.getSessionFactory();
Session session = factory.openSession();
try{
tcx = session.beginTransaction();
session.save(pBlindStructure);
tcx.commit();
}
catch( Throwable e){
tcx.rollback();
}
finally{
session.close();
}
return pBlindStructure.getIdBlindStructure();
}
It correctly saved my object.
I test one more case :
Just returning a constant and don't put Id in the variable like first example and It work. It seems that object is not save in case I get the ID directly with session.save() method.
Moreover, I observe something interesting. I made a first test with one of the solution which worked, it generated a database data with Id 117. Then I changed my code for the solution which difsn't work and reloaded it in Tomcat , I made 2nd try without success. I changed again my code for the one which succeded and the id was generated is 120. It missed 2nf id number (the 2nd try I've done ??)
To help you see my hibernate.cfg.xml file
<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
'-//Hibernate/Hibernate Configuration DTD 3.0//EN'
'http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd'>
<hibernate-configuration>
<session-factory>
<!-- Database connection settings -->
<property name='connection.driver_class'>com.mysql.jdbc.Driver</property>
<property name='connection.url'>jdbc:mysql://XXXXXX:XXXX/PokerLeagueManager</property>
<property name='connection.username'>XXXXX</property>
<property name='connection.password'>XXXXXX</property>
<property name="show_sql">true</property>
<!-- JDBC connection pool (use the built-in) -->
<property name="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.acquire_increment">1</property>
<property name="hibernate.c3p0.idle_test_period">120</property>
<property name="hibernate.c3p0.min_size">1</property>
<property name="hibernate.c3p0.max_size">10</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.timeout">120</property>
<property name="hibernate.c3p0.acquireRetryAttempts">1</property>
<property name="hibernate.c3p0.acquireRetryDelay">250</property>
<!-- Dev -->
<property name="hibernate.c3p0.validate">true</property>
<!-- SQL dialect -->
<property name='dialect'>org.hibernate.dialect.MySQL5InnoDBDialect</property>
<!-- Enable Hibernate's automatic session context management -->
<property name="current_session_context_class">thread</property>
<!-- Disable the second-level cache -->
<property name="cache.provider_class">org.hibernate.cache.internal.NoCacheProvider</property>
<!-- Echo all executed SQL to stdout -->
<property name='show_sql'>true</property>
<mapping resource="mappings/BlindStructure.hbm.xml"/>
<mapping resource="mappings/Tournament.hbm.xml"/>
<mapping resource="mappings/LegalFee.hbm.xml"/>
</session-factory>
Edit: User3813463 answer for a part of a question. It explains that session are in AUTO flush mode by default which flush session in some case like :
flush occurs by default at the following points:
before some query executions
from org.hibernate.Transaction.commit()
from Session.flush()
But I my point of view (I maybe miss understand something), my first case should worked because I commit my transaction.
Secondly, I need advise to choose a flush mode. To my mind Commit mode is a good way to do for method which make only inserting or only reading data on database.
Do you agree with me, have you some sources where this is debate?

This all is the magic of FlushMode read docs here
According to docs default FlushMode is AUTO that means
The Session is sometimes flushed before query execution in order to
ensure that queries never return stale state.
According to another document (Here):
flush occurs by default at the following points:
before some query executions
from org.hibernate.Transaction.commit()
from Session.flush()
So when you say pBlindStructure.getIdBlindStructure(); hibernate actually perform flush on current session, resulting data is getting saved in DB.

The session save method returns an object for the generated id by the generator. This value you should cast to Long.
long id = -1;
try{
tcx = session.beginTransaction();
id = (Long) session.save(pBlindStructure);
tcx.commit();
}
catch( Throwable e){
tcx.rollback();
}
finally{
session.close();
}
return id;

How to run native SQL queries in the same Hibernate transaction?

We have a Service which is #Stateful. Most of the Data-Operations are atomic, but within a certain set of functions We want to run multiple native queries within one transaction.
We injected the EntityManager with a transaction scoped persistence context. When creating a "bunch" of normal Entities, using em.persist() everything is working fine.
But when using native queries (some tables are not represented by any #Entity) Hibernate does not run them within the same transaction but basically uses ONE transaction per query.
So, I already tried to use manual START TRANSACTION; and COMMIT; entries - but that seems to interfere with the transactions, hibernate is using to persist Entities, when mixing native queries and persistence calls.
#Stateful
class Service{
#PersistenceContext(unitName = "service")
private EntityManager em;
public void doSth(){
this.em.createNativeQuery("blabla").executeUpdate();
this.em.persist(SomeEntity);
this.em.createNativeQuery("blablubb").executeUpdate();
}
}
Everything inside this method should happen within one transaction. Is this possible with Hibernate?
When debugging it, it is clearly visible that every statement happens "independent" of any transaction. (I.e. Changes are flushed to the database right after every statement.)
I've tested the bellow given example with a minimum setup in order to eliminate any other factors on the problem (Strings are just for breakpoints to review the database after each query):
#Stateful
#TransactionManagement(value=TransactionManagementType.CONTAINER)
#TransactionAttribute(value=TransactionAttributeType.REQUIRED)
public class TestService {
#PersistenceContext(name = "test")
private EntityManager em;
public void transactionalCreation(){
em.createNativeQuery("INSERT INTO `ttest` (`name`,`state`,`constraintCol`)VALUES('a','b','c')").executeUpdate();
String x = "test";
em.createNativeQuery("INSERT INTO `ttest` (`name`,`state`,`constraintCol`)VALUES('a','c','b')").executeUpdate();
String y = "test2";
em.createNativeQuery("INSERT INTO `ttest` (`name`,`state`,`constraintCol`)VALUES('c','b','a')").executeUpdate();
}
}
Hibernate is configured like this:
<persistence-unit name="test">
<provider>org.hibernate.jpa.HibernatePersistenceProvider</provider>
<jta-data-source>java:jboss/datasources/test</jta-data-source>
<properties>
<property name="hibernate.dialect" value="org.hibernate.dialect.MySQL5InnoDBDialect" />
<property name="hibernate.transaction.jta.platform"
value="org.hibernate.service.jta.platform.internal.JBossAppServerJtaPlatform" />
<property name="hibernate.archive.autodetection" value="true" />
<property name="hibernate.jdbc.batch_size" value="20" />
<property name="connection.autocommit" value="false"/>
</properties>
</persistence-unit>
And the outcome is the same as with autocommit mode: After every native query, the database (reviewing content from a second connection) is updated immediately.
The idea of using the transaction in a manual way leads to the same result:
public void transactionalCreation(){
Session s = em.unwrap(Session.class);
Session s2 = s.getSessionFactory().openSession();
s2.setFlushMode(FlushMode.MANUAL);
s2.getTransaction().begin();
s2.createSQLQuery("INSERT INTO `ttest` (`name`,`state`,`constraintCol`)VALUES('a','b','c')").executeUpdate();
String x = "test";
s2.createSQLQuery("INSERT INTO `ttest` (`name`,`state`,`constraintCol`)VALUES('a','c','b')").executeUpdate();
String y = "test2";
s2.createSQLQuery("INSERT INTO `ttest` (`name`,`state`,`constraintCol`)VALUES('c','b','a')").executeUpdate();
s2.getTransaction().commit();
s2.close();
}

In case you don't use container managed transactions then you need to add the transaction policy too:
#Stateful
#TransactionManagement(value=TransactionManagementType.CONTAINER)
#TransactionAttribute(value=REQUIRED)
I have only seen this phenomenon in two situations:
the DataSource is running in auto-commit mode, hence each statement is executed in a separate transaction
the EntityManager was not configured with #Transactional, but then only queries can be run since any DML operation would end-up throwing a transaction required exception.
Let's recap you have set the following Hibernate properties:
hibernate.current_session_context_class=JTA
transaction.factory_class=org.hibernate.transaction.JTATransactionFactory
jta.UserTransaction=java:comp/UserTransaction
Where the final property must be set with your Application Server UserTransaction JNDI naming key.
You could also use the:
hibernate.transaction.manager_lookup_class=org.hibernate.transaction.JBossTransactionManagerLookup
or some other strategy according to your current Java EE Application Server.

After reading about the topic for another bunch of hours while playing around with every configuration property and/or annotation I could find a working solution for my usecase. It might not be the best or only solution, but since the question has received some bookmarks and upvotes, i'd like to share what i have so far:
At first, there was no way to get it working as expected when running the persistence-unit in managed mode. (<persistence-unit name="test" transaction-type="JTA"> - JTA is default if no value given.)
I decided to add another persistence-unit to the persistence xml, which is configured to run in unmanaged mode: <persistence-unit name="test2" transaction-type="RESOURCE_LOCAL">.
(Note: The waring about Multiple Persistence Units is just cause eclipse can't handle. It has no functional impact at all)
The unmanaged persitence-context requires local configuration of the database, since it is no longer container-provided:
<persistence-unit name="test2" transaction-type="RESOURCE_LOCAL">
<provider>org.hibernate.jpa.HibernatePersistenceProvider</provider>
<class>test.AEntity</class>
<properties>
<property name="hibernate.connection.url" value="jdbc:mysql://localhost/test"/>
<property name="hibernate.dialect" value="org.hibernate.dialect.MySQL5InnoDBDialect" />
<property name="hibernate.connection.driver_class" value="com.mysql.jdbc.Driver"/>
<property name="hibernate.connection.password" value="1234"/>
<property name="hibernate.connection.username" value="root"/>
<property name="hibernate.hbm2ddl.auto" value="update" />
<property name="hibernate.show_sql" value="true" />
<property name="hibernate.archive.autodetection" value="true" />
<property name="hibernate.jdbc.batch_size" value="20" />
<property name="hibernate.connection.autocommit" value="false" />
</properties>
</persistence-unit>
A change required to the project would now be, that you add an unitName, whenever you use the #PersistenceContext annotation to retrieve a managed instance of the EntityManager.
But be aware, that you can only use #PersistenceContext for the managed persistence-unit. For the unmanaged one, you could implement a simple Producer and Inject the EntityManager using CDI whenever required:
#ApplicationScoped
public class Resources {
private static EntityManagerFactory emf;
static {
emf = Persistence.createEntityManagerFactory("test2");
}
#Produces
public static EntityManager createEm(){
return emf.createEntityManager();
}
}
Now, in the example given in the original Post, you need to Inject the EntityManager and manually take care about transactions.
#Stateful
public class TestService {
#Inject
private EntityManager em;
public void transactionalCreation() throws Exception {
em.getTransaction().begin();
try {
em.createNativeQuery(
"INSERT INTO `ttest` (`name`,`state`,`constraintCol`)VALUES('a','b','a')")
.executeUpdate();
em.createNativeQuery(
"INSERT INTO `ttest` (`name`,`state`,`constraintCol`)VALUES('a','b','b')")
.executeUpdate();
em.createNativeQuery(
"INSERT INTO `ttest` (`name`,`state`,`constraintCol`)VALUES('a','b','c')")
.executeUpdate();
em.createNativeQuery(
"INSERT INTO `ttest` (`name`,`state`,`constraintCol`)VALUES('a','b','d')")
.executeUpdate();
AEntity a = new AEntity();
a.setName("TestEntity1");
em.persist(a);
// force unique key violation, rollback should appear.
// em.createNativeQuery(
// "INSERT INTO `ttest` (`name`,`state`,`constraintCol`)VALUES('a','b','d')")
// .executeUpdate();
em.getTransaction().commit();
} catch (Exception e) {
em.getTransaction().rollback();
}
}
}
My tests so far showed that mixing of native queries and persistence calls lead to the desired result: Either everything is commited or the transaction is rolledback as a whole.
For now, the solution seems to work. I will continue to validate it's functionality in the main project and check if there are any other sideeffects.
Another thing I need to verify is if it would be save to:
Inject both Versions of the EM into one Bean and mix usage. (First checks seem to work, even when using both ems at the same time on the same table(s))
Having both Versions of the EM operating on the same datasource. (Same data source would most likely be no problem, same tables I assume could lead to unexpected problems.)
ps.: This is Draft 1. I will continue to improve the answer and point out problems and/or drawbacks I'm going to find.

You have to add <hibernate.connection.release_mode key="hibernate.connection.release_mode" value="after_transaction" /> to your properties. After a restart should the Transaction handling work.

Save embedded HSQL database to file

For logging and debugging purposes I want to dump an embedded/in-memory HSQL database to a file. Schema + Data. I'm using the spring-framework with hibernate.
I've tried both:
Session session = sessionFactory.openSession();
Transaction transaction = session.beginTransaction();
session.createSQLQuery("BACKUP DATABASE TO '/tmp/backup.tar.gz' BLOCKING");
transaction.commit();
session.close();
and
Session session = sessionFactory.openSession();
Transaction transaction = session.beginTransaction();
session.createSQLQuery("SCRIPT '/tmp/backup-data.sql'");
transaction.commit();
session.close();
Both to no avail.
There is no special configuration.
Hibernate config:
<jdbc:embedded-database id="dataSource" type="HSQL">
<jdbc:script location="classpath:spring/batch/database/schema.sql"/>
</jdbc:embedded-database>
<bean id="sessionFactory"
class="org.springframework.orm.hibernate4.LocalSessionFactoryBean"
p:dataSource-ref="dataSource">
<property name="packagesToScan" value="com.domain.*.model"/>
<property name="hibernateProperties">
<value>
hibernate.format_sql=true
hibernate.dialect=org.hibernate.dialect.HSQLDialect
</value>
</property>
</bean>
What did I miss, do I need a different approach, is it even possible?

Your code is only creating an instance of a SQLQuery and throws it away. You aren't executing the query.
Add/change to the following
SQLQuery query = session.createSQLQuery("BACKUP DATABASE TO '/tmp/backup.tar.gz' BLOCKING");
query.executeUpdate();
This will execute the query.

Have you tried:
Session session = sessionFactory.openSession();
Transaction transaction = session.beginTransaction();
session.createSQLQuery("BACKUP DATABASE TO '/tmp/' BLOCKING");
transaction.commit();
session.close();
According to the documentation you need to give the directory and HSQL will create the tar.gz file in it.

Batch Insertions with Hibernate & Spring

My application is based on Hibernate 3.2 and Spring 2.5. Here is the transaction management related snippet from the application context:
<tx:annotation-driven transaction-manager="txManager"/>
<bean id="txManager" class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory" ref="sessionFactory"/>
<property name="nestedTransactionAllowed" value="true"/>
</bean>
<bean id="transactionTemplate" classs="org.springframework.transaction.support.TransactionTemplate">
<property name="transactionManager" ref="txManager"/>
</bean>
<bean class="org.springframework.beans.factory.annotation.RequiredAnnotationBeanPostProcessor"/>
<bean id="sessionFactory" class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
<property name="configLocation" value="classpath:/hibernate.cfg.xml"></property>
</bean>
For all the DAO's there are relevant Service class and the transactions are handled there using #Transactional on each method in the service layer. However there is a scenario now that a method in DAO say "parse()" is called from the service layer. In the service layer I specified #Transactional(readOnly=false). This parse method in the DAO calls another method say "save()" in the same DAO which stores a large number of rows (around 5000) in the database. Now the save method is called in a loop from the parse function. Now the issue is that after around 100 calls to the "save" method.. i sometimes get a OutOfMemory Exception or sometimes the program stops responding.
For now these are the changes which I have made to the save method:
Session session = getHibernateTemplate().getSessionFactory().openSession();
Transaction tx = session.beginTransaction();
int counter = 0;
if(books!=null && !books.isEmpty()){
for (Iterator iterator = books.iterator(); iterator
.hasNext();) {
Book book = (Book) iterator.next();
session.save(book);
counter++;
if(counter % 20==0) {
session.flush();
session.clear();
}
}
}
tx.commit();
session.close();
This is the only method in my application where I start a transaction like this and commit it at the end of method. Otherwise I normally just call getHibernateTemplate.save(). I am not sure whether I should perform transaction management for this save method separately in the DAO by placing #Transactional(readOnly=false, PROPOGATION=NEW) on save(), or is this approach okay?
Also I have updated the hibernate.jdbc.batch_size to 20 in the hibernate.cfg configuration file.
Any suggestions?

For the batch insertion with hibernate, the best practice is StatelessSession, it doesn`t cache any states of your entity, you will not encounter OutOfMemory, the code like:
if (books == null || books.isEmpty) {
return;
}
StatelessSession session = getHibernateTemplate().getSessionFactory().openStatelessSession();
Transaction tx = session.beginTransaction();
for (Book each : books) {
session.insert(book);
}
tx.commit();
session.close();
And the Transaction of StatelessSession is independent from the current transaction context.

You only need the bit with flushing and clearing the session. Leave transaction management to Spring. Use sessionFactory.getCurrentSession() to reach the session that Spring has already opened for you. Also, Spring's recent recommmendation is to avoid HibernateTemplate and work directly with Hibernate's API. Inject SessionFactory to your dao-bean.

I would refactor parse in a way it doesn't call save directly but takes some callback from service layer. Service layer would pass its transactional method with save call as this callback.
It may not work exactly as decribed in your case but from this short description this would be something I'd try.