Develop Reference

Spring Data JPA Additional EntityManagerFactory Optimized for Cache and Bulk Operations Only

I have a legacy Spring Data JPA application that has a large number of Entities and CrudRepositories. JPA is configured using the XML below. We have a new requirement that requires us to insert 10,000 - 50,000 entities into the database at once via a FileUpload. With the existing configuration the database CPU spikes. After enabling hibernate statistics, its was apparent that these 10,000 insert operations were generating over 200,000 DB queries due to all the validation logic required with a single insert in the InvoiceService.
Original Configuration
<bean id="dataSource" destroy-method="close" class="org.apache.commons.dbcp2.BasicDataSource">
<property name="driverClassName" value="${db.driver}"/>
<property name="url" value="${db.jdbcurl}"/>
<property name="username" value="${db.username}"/>
<property name="password" value="${db.password}"/>
<property name="maxTotal" value="${db.maxTotal}"/>
<property name="maxIdle" value="${db.maxIdle}"/>
<property name="minIdle" value="${db.minIdle}"/>
<property name="initialSize" value="${db.initialSize}"/>
<property name="maxWaitMillis" value="${db.maxWaitMillis}"/>
<property name="minEvictableIdleTimeMillis" value="${db.minEvictableIdleTimeMillis}"/>
<property name="timeBetweenEvictionRunsMillis" value="${db.timeBetweenEvictionRunsMillis}"/>
<property name="testOnBorrow" value="${db.testOnBorrow}"/>
<property name="testOnReturn" value="${db.testOnReturn}"/>
<property name="testWhileIdle" value="${db.testWhileIdle}"/>
<property name="removeAbandonedOnBorrow" value="${db.removeAbandonedOnBorrow}"/>
<property name="removeAbandonedOnMaintenance" value="${db.removeAbandonedOnMaintenance}"/>
<property name="removeAbandonedTimeout" value="${db.removeAbandonedTimeout}"/>
<property name="logAbandoned" value="${db.logAbandoned}"/>
</bean>
<bean id="entityManagerFactory"
class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean" depends-on="flyway">
<property name="dataSource" ref="dataSource" />
<property name="packagesToScan" value="my.package.domain" />
<property name="loadTimeWeaver">
<bean class="org.springframework.instrument.classloading.InstrumentationLoadTimeWeaver" />
</property>
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter" />
</property>
<property name="jpaProperties">
<props>
<prop key="hibernate.hbm2ddl.auto">${hibernate.hbm2ddl.auto:validate}</prop>
<prop key="hibernate.dialect">org.hibernate.dialect.PostgreSQL9Dialect</prop>
<prop key="hibernate.show_sql">${hibernate.show_sql:false}</prop>
</props>
</property>
<property name="persistenceUnitName" value="entityManagerFactory" />
</bean>
<bean id="persistenceAnnotationBeanPostProcessor" class="org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor">
<property name="defaultPersistenceUnitName" value="entityManagerFactory"/>
</bean>
<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactory" />
</bean>
<tx:annotation-driven proxy-target-class="true" />
<bean id="persistenceExceptionTranslationPostProcessor"
class="org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor" />
<jpa:repositories base-package="my.package.repository" entity-manager-factory-ref="entityManagerFactory"/>
The FileUploadService snippet looks like this...
EntityManager batchEntityManager = entityManagerFactory.createEntityManager();
EntityTransaction transaction = batchEntityManager.getTransaction();
transaction.begin();
try (BufferedReader buffer = new BufferedReader(new InputStreamReader(file.getInputStream()))) {
buffer.lines()
.filter(StringUtils::isNotBlank)
.forEach(csvLine -> {
invoiceService.createInvoice(csvLine);
if (counter.incrementAndGet() % (updateFrequency.equals(0) ? 1 : updateFrequency) == 0) {
FileUpload fileUpload1 = fileUploadRepository.findOne(fileUpload.getId());
fileUpload1.setNumberOfSentRecords(sentCount.get());
fileUploadRepository.save(fileUpload1);
transaction.commit();
transaction.begin();
batchEntityManager.clear();
}
});
transaction.commit();
} catch (IOException ex) {
systemException.incrementAndGet();
log.error("Unexpected error while performing send task.", ex);
transaction.rollback();
}
// Update FileUpload status.
FileUpload fileUpload1 = fileUploadRepository.findOne(fileUpload.getId());
fileUpload1.setNumberOfSentRecords(sentCount.get());
if (systemException.get() != 0) {
fileUpload1.setStatus(FileUploadStatus.SYSTEM_ERROR);
} else {
fileUpload1.setStatus(FileUploadStatus.SENT);
}
fileUploadRepository.save(fileUpload1);
batchEntityManager.close();
Most of the DB queries were select statements that return the same results for each record being inserted. It was obvious that enabling EhCache as a Second-Level cache would have a significant performance improvement. However, this application has been running flawlessly in production for several years without ehcache enabled. I am hesitant to turn this on globally as I do not know how this will affect the large number of other repositories/queries.
Question 1
Is there a way to configure a "alternate" EntityManagerFactory that uses the second level cache for this batch process only? How can I choose to use this factory instead of the primary for this batch process only?
I experimented adding something like below to my spring config. I can easily inject this additional EntityManager into my class and use it. However, the existing Repositories (such as FileUploadRepository) don't seem to use it - they just return null. I am not sure if this approach is possible. The documentation for JpaTransactionManager says
This transaction manager is appropriate for applications that use a single JPA EntityManagerFactory for transactional data access
which is exactly what I am not doing. So what other options are there?
<bean id="batchEntityManagerFactory"
class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean" depends-on="flyway">
<property name="dataSource" ref="dataSource" />
<property name="packagesToScan" value="my.package.domain" />
<property name="loadTimeWeaver">
<bean class="org.springframework.instrument.classloading.InstrumentationLoadTimeWeaver" />
</property>
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter" />
</property>
<property name="jpaProperties">
<props>
<prop key="hibernate.hbm2ddl.auto">${hibernate.hbm2ddl.auto:validate}</prop>
<prop key="hibernate.dialect">org.hibernate.dialect.PostgreSQL9Dialect</prop>
<prop key="hibernate.show_sql">${hibernate.show_sql:false}</prop>
<prop key="hibernate.generate_statistics">${hibernate.generate_statistics:false}</prop>
<prop key="hibernate.ejb.interceptor">my.package.HibernateStatistics</prop>
<prop key="hibernate.cache.use_query_cache">true</prop>
<prop key="hibernate.cache.use_second_level_cache">true</prop>
<prop key="hibernate.cache.region.factory_class">org.hibernate.cache.ehcache.EhCacheRegionFactory</prop>
<prop key="hibernate.jdbc.batch_size">100</prop>
<prop key="hibernate.order_inserts">true</prop>
<prop key="hibernate.order_updates">true</prop>
</props>
</property>
<property name="persistenceUnitName" value="entityManagerFactory" />
</bean>
Question 2
Assuming there is no other option to "selectively" use EhCache, I tried enabling it on the primary EntityManagerFactory only. We can certainly do regression testing to make sure we don't introduce new issues. I assume this is fairly safe to do? However, another issue came up. I am trying to commit inserts in batches as described in this post and shown in my code above. I am getting an RollbackException when trying to commit the batch due to Connection org.postgresql.jdbc.PgConnection#1e7eb804 is closed.. I presume this is due to the maxWaitMillis property on the dataSource.
I don't want to change this property for every other existing spring Service/Repository/query in the application. If I could use the "custom" EntityManagerFactory I could easily provide a different DataSource Bean with the properties I want. Again, is this possible?
Maybe I looking at this problem all wrong. Are there any other suggestions?

You can have another EntityManagerFactory bean with a different qualifier, so that's one option. I would still recommend you look into these select queries. I bet the problem is just a missing index which causes the database to do full table scans. If you add the proper indexes, you probably don't need to change a thing in your application.

How to configure Hibernate 5 SessionFactoryBean for using HikariCP?

Where should I configure connection pool for spring + hibernate application?
<bean id="sessionFactory" class="org.springframework.orm.hibernate5.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="packagesToScan" value="edu.khai.education.entity"/>
<property name="hibernateProperties">
<props>
<prop key="hibernate.show_sql">true</prop>
<prop key="hibernate.dialect">org.hibernate.dialect.H2Dialect</prop>
</props>
</property>
</bean>
<bean id="dataSource" class="org.apache.commons.dbcp2.BasicDataSource">
<property name="driverClassName" value="org.h2.Driver"/>
<property name="url" value="jdbc:h2:tcp://localhost/~/test"/>
<property name="username" value="sa"/>
<property name="password" value=""/>
</bean>
<bean id="txManager" class="org.springframework.orm.hibernate5.HibernateTransactionManager">
<property name="sessionFactory" ref="sessionFactory"/>
</bean>
I want to use org.hibernate.hikaricp.internal.HikariCPConnectionProvider and I don't have a problem to configure it in Hibernate application, but I don't know how to integrate it with Spring.

Spring uses DataSource bean to create a lot of infrastructure code e.g. JdbcTemplate or Spring Data JPA #Repository.
Replace the current data source implementation org.apache.commons.dbcp2.BasicDataSource with the com.zaxxer.hikari.HikariDataSource class. Change the current properties to match the HikariCP configuration.

Error No Session found for current thread when trying to save into two database simultaneously

Hi i'm trying to save into two database simultaneously but I always get an error.
Exception in thread "main" org.hibernate.HibernateException: No Session found for current thread
here's my code :
#Transactional(rollbackFor=Exception.class, propagation=Propagation.REQUIRES_NEW, readOnly=false)
public void save(ArsenalPlayer domain1, ArsenalPlayer2 domain2)
throws Exception {
dao1.save(domain1);
dao2.save(domain2);
}
dao1 uses a sessionFactory attached to datasource1
dao2 uses a sessionFactory attached to datasource2
and here's my configuration
DataSource
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource">
<property name="driverClass" value="com.mysql.jdbc.Driver" />
<property name="jdbcUrl" value="jdbc:mysql://localhost:3306/arsenal" />
<property name="user" value="root" />
<property name="password" value="ahmids" />
</bean>
SessionFactory
<bean id="sessionFactory"
class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.MySQLDialect</prop>
<prop key="hibernate.show_sql">true</prop>
<prop key="hibernate.format_sql">update</prop>
<prop key="hibernate.hbm2ddl.auto">update</prop>
</props>
</property>
<property name="annotatedClasses">
<list>
<value>com.gongfu4.bean.ArsenalPlayer</value>
</list>
</property>
</bean>
DataSource2
<bean id="dataSource2" class="com.mchange.v2.c3p0.ComboPooledDataSource">
<property name="driverClass" value="com.mysql.jdbc.Driver" />
<property name="jdbcUrl" value="jdbc:mysql://localhost:3306/arsenal2" />
<property name="user" value="root" />
<property name="password" value="ahmids" />
</bean>
SessionFactory2
<bean id="sessionFactory2"
class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource2" />
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.MySQLDialect</prop>
<prop key="hibernate.show_sql">true</prop>
<prop key="hibernate.format_sql">update</prop>
<prop key="hibernate.hbm2ddl.auto">update</prop>
<!-- <prop key="hibernate.current_session_context_class">thread</prop> -->
</props>
</property>
<property name="annotatedClasses">
<list>
<value>com.gongfu4.bean.ArsenalPlayer2</value>
</list>
</property>
</bean>
applicationContext
<context:component-scan base-package="com.gongfu4" />
<context:annotation-config />
<import resource="dataSource.xml" />
<import resource="dataSource2.xml" />
<import resource="hibernate.xml" />
<import resource="hibernate2.xml" />
<tx:annotation-driven />
<bean id="transactionManager"
class="org.springframework.orm.hibernate4.HibernateTransactionManager">
<property name="sessionFactory" ref="sessionFactory" />
</bean>
<bean id="myTx"
class="org.springframework.orm.hibernate4.HibernateTransactionManager">
<property name="sessionFactory" ref="sessionFactory2" />
</bean>
and here are my dao classes :
DAO1
#Autowired
SessionFactory sessionFactory;
public void save(ArsenalPlayer domain) {
sessionFactory.getCurrentSession().save(domain);
}
DAO2
#Autowired
SessionFactory sessionFactory2;
public void save(ArsenalPlayer2 domain) {
sessionFactory2.getCurrentSession().merge(domain);
}
Is there something wrong with my configuration?

You cannot use those two transaction managers like that. You have two data sources, two transaction managers and, from your code, I understand that you want the two save operations to be performed in the same transaction. The question is "a transaction from which transaction manager?".
The way you have the code and the configuration, Spring will use the "default" transaction manager because <tx:annotation-driven/> by default will search for a transaction manager bean with the id "transactionManager" and the bean with this id is the one for data source dataSource. But your code is not working and this is the expected behavior. Spring will open a Hibernate session using sessionFactory and the dao1.save(domain1); call will succeed because this is the correct Hibernate session for the correct dataSource. But when the dao2.save(domain2); method is called you will have the same session from dao1 call but be used for a db operation for the second database.
As I see it you have two options:
Use a JTA transaction manager to coordinate the two datasources. With a JTA the two save operations will be atomic. If one fails, then both are rolled back.
Perform the two save(domain) operations in two different transactions properly configuring the #Transactional annotation to use the correct transaction manager. In this case, the two save operations will not be atomic. If one save fails then only that one will be rolled-back. See below an exempt taken from the Spring reference documentation here:
public class TransactionalService {
#Transactional("order")
public void setSomething(String name) { ... }
#Transactional("account")
public void doSomething() { ... }
}
<tx:annotation-driven/>
<bean id="transactionManager1" class="org.springframework.jdbc.DataSourceTransactionManager">
...
<qualifier value="order"/>
</bean>
<bean id="transactionManager2" class="org.springframework.jdbc.DataSourceTransactionManager">
...
<qualifier value="account"/>
</bean>

Your current configuration doesn't support global transactions (XA) and therefore you can't span one Transaction on two different databases.
For this reason you need two Hiberante transaction managers, one for each session factory. Then you need to instruct the transactional service which transaction manager should be used.
So instead of:
#Transactional(rollbackFor=Exception.class, propagation=Propagation.REQUIRES_NEW, readOnly=false)
public void save(ArsenalPlayer domain1, ArsenalPlayer2 domain2)
throws Exception {
dao1.save(domain1);
dao2.save(domain2);
}
you should have:
<bean id="txManager1" class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory" ref="sessionFactory1" />
</bean>
<bean id="txManager2" class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory" ref="sessionFactory2" />
</bean>
#Transactional(rollbackFor=Exception.class, propagation=Propagation.REQUIRES_NEW, readOnly=false, value="txManager1")
public void save(ArsenalPlayer domain1)
throws Exception {
dao1.save(domain1);
}
#Transactional(rollbackFor=Exception.class, propagation=Propagation.REQUIRES_NEW, readOnly=false, value="txManager2")
public void save(ArsenalPlayer domain2)
throws Exception {
dao2.save(domain2);
}

Correctly setting up JTA Transaction Manager in Tomcat Server with Atomikos

How would I setup container-managed datasources and embedded Active MQ resources to JTATransactionManager for global Transactions?
I am using Tomcat 6 and installed Atomikos in it to support JTA. I use Hibernate for ORM. Here is my configuration:
<bean id="AtomikosTransactionManager"
class="com.atomikos.icatch.jta.UserTransactionManager"
init-method="init" destroy-method="close">
<!-- when close is called, should we force
transactions to terminate or not? -->
<property name="forceShutdown" value="false" />
</bean>
<bean id="AtomikosUserTransaction"
class="com.atomikos.icatch.jta.UserTransactionImp">
<property name="transactionTimeout" value="300" />
</bean>
<jee:jndi-lookup expected-type="javax.sql.DataSource" id="dataSource" jndi-name="jdbc/EDITSOLUTIONS"/>
<bean id="sessionFactory" class="org.springframework.orm.hibernate.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="mappingResources"/>
<list>
<value>../../src/editsolutions.hibernate.cfg.xml</value>
</list>
<property name="hibernateProperties">
<value>
hibernate.dialect=${hibernate.dialect}
</value>
</property>
</bean>
<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="connectionFactory">
<bean class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="failover://tcp://localhost:61616"/>
</bean>
</property>
</bean>
<bean id="jmsConnectionFactory"class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="failover://tcp://localhost:61616"/>
</bean>
<bean name="txManager" class="org.springframework.transaction.jta.JtaTransactionManager">
<property name="transactionManager" ref="AtomikosTransactionManager" />
<property name="userTransaction" ref="AtomikosUserTransaction" />
</bean>
Spring documentation says that JTA Transaction Manager need not be told about resources. That's what I have done.
I have the following outstanding questions:
I am not sure whether Atomikos is integrated properly or not?
Is it OK to get the datasource from <jee:jndi-lookup>?
Is Hiberante configuration correct with respect to JTATransactionManager?
As it is embedded in JVM not managed by container, would JTATransactionManager be able to recognize ActimeMQ ?

Try with this very useful link: http://www.atomikos.com/Documentation/SpringIntegration
Remember to configure the datasource in this way:
<bean id="dataSourceA" class="com.atomikos.jdbc.AtomikosDataSourceBean" init-method="init" destroy-method="close">
<qualifier value="jmsRecoveryDatabaseSchema"/>
<property name="uniqueResourceName"><value>XADS1</value></property>
<property name="xaDataSourceClassName">
<value>oracle.jdbc.xa.client.OracleXADataSource</value>
</property>
<property name="xaProperties">
<props>
<prop key="URL">${jdbc.url}</prop>
<prop key="user">${jdbc.username}</prop>
<prop key="password">${jdbc.password}</prop>
</props>
</property>
<property name="poolSize"><value>1</value></property>
<property name="testQuery" value="SELECT 1 FROM DUAL"/>
</bean>
I hope it helps!

No Hibernate session bound to the thread while using multiple transation managers

I have created two transaction managers(one for READ requests and another for READ-WRITE requests) and two session factories for the same.
Important snippets are as follows:
<tx:annotation-driven />
<bean id="txManager"
class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory" ref="SessionFactory" />
</bean>
<bean id="txManagerRead"
class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory" ref="SessionFactory" />
<qualifier value="READ"/>
</bean>
<bean id="txManagerWrite"
class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory" ref="SessionFactory1" />
<qualifier value="WRITE"/>
</bean>
<bean id="AbstractSessionFactory"
class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean"
abstract="true">
<property name="annotatedClasses">
<list>
.....
....
</list>
</property>
<property name="exposeTransactionAwareSessionFactory">
<value>true</value>
</property>
</bean>
<bean id="SessionFactory"
class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean"
parent="AbstractSessionFactory">
<property name="hibernateProperties">
<props>
.........
.........
</props>
</property>
</bean>
<bean id="SessionFactory1"
class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean"
parent="AbstractSessionFactory">
<property name="hibernateProperties">
<props>
.........
.........
</props>
</property>
</bean>
I am getting following exception :
Caused by: org.hibernate.HibernateException: No Hibernate Session bound to thread, and configuration does not allow creation of non-transactional one here
at org.springframework.orm.hibernate3.SpringSessionContext.currentSession(SpringSessionContext.java:63)
at org.hibernate.impl.SessionFactoryImpl.getCurrentSession(SessionFactoryImpl.java:574)
at com.amazon.im.dao.hibernate.GenericDAOImpl.getSession(GenericDAOImpl.java:53)
at com.amazon.im.dao.hibernate.CategoryDAOImpl.findAllActiveCategories(CategoryDAOImpl.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Method which calls DAO is as follows:
#Transactional(rollbackFor = { DependencyException.class, ServiceException.class }, readOnly = true, value="READ")
#Operation("getActiveCategories")
public GetActiveCategoriesResponse getActiveCategories() throws ServiceException, DependencyException
{ .....
......
......
}
Can anyone point me to the part I am doing wrong?

Try using #Transactional("Tx_Manager_Name") as per this . Your configuration seems to be correct since multiple Txn managers are supported through Qualifiers as well .

Your configuration is wrong. Hibernate and even spring allows you to create only one session factory per app. Check out this link : Spring + Hibernate session lifecycle. You can create session per transaction.
Also check this link : http://docs.jboss.org/hibernate/core/3.5/api/org/hibernate/SessionFactory.html