Develop Reference

Spring Data JPA Additional EntityManagerFactory Optimized for Cache and Bulk Operations Only

I have a legacy Spring Data JPA application that has a large number of Entities and CrudRepositories. JPA is configured using the XML below. We have a new requirement that requires us to insert 10,000 - 50,000 entities into the database at once via a FileUpload. With the existing configuration the database CPU spikes. After enabling hibernate statistics, its was apparent that these 10,000 insert operations were generating over 200,000 DB queries due to all the validation logic required with a single insert in the InvoiceService.
Original Configuration
<bean id="dataSource" destroy-method="close" class="org.apache.commons.dbcp2.BasicDataSource">
<property name="driverClassName" value="${db.driver}"/>
<property name="url" value="${db.jdbcurl}"/>
<property name="username" value="${db.username}"/>
<property name="password" value="${db.password}"/>
<property name="maxTotal" value="${db.maxTotal}"/>
<property name="maxIdle" value="${db.maxIdle}"/>
<property name="minIdle" value="${db.minIdle}"/>
<property name="initialSize" value="${db.initialSize}"/>
<property name="maxWaitMillis" value="${db.maxWaitMillis}"/>
<property name="minEvictableIdleTimeMillis" value="${db.minEvictableIdleTimeMillis}"/>
<property name="timeBetweenEvictionRunsMillis" value="${db.timeBetweenEvictionRunsMillis}"/>
<property name="testOnBorrow" value="${db.testOnBorrow}"/>
<property name="testOnReturn" value="${db.testOnReturn}"/>
<property name="testWhileIdle" value="${db.testWhileIdle}"/>
<property name="removeAbandonedOnBorrow" value="${db.removeAbandonedOnBorrow}"/>
<property name="removeAbandonedOnMaintenance" value="${db.removeAbandonedOnMaintenance}"/>
<property name="removeAbandonedTimeout" value="${db.removeAbandonedTimeout}"/>
<property name="logAbandoned" value="${db.logAbandoned}"/>
</bean>
<bean id="entityManagerFactory"
class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean" depends-on="flyway">
<property name="dataSource" ref="dataSource" />
<property name="packagesToScan" value="my.package.domain" />
<property name="loadTimeWeaver">
<bean class="org.springframework.instrument.classloading.InstrumentationLoadTimeWeaver" />
</property>
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter" />
</property>
<property name="jpaProperties">
<props>
<prop key="hibernate.hbm2ddl.auto">${hibernate.hbm2ddl.auto:validate}</prop>
<prop key="hibernate.dialect">org.hibernate.dialect.PostgreSQL9Dialect</prop>
<prop key="hibernate.show_sql">${hibernate.show_sql:false}</prop>
</props>
</property>
<property name="persistenceUnitName" value="entityManagerFactory" />
</bean>
<bean id="persistenceAnnotationBeanPostProcessor" class="org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor">
<property name="defaultPersistenceUnitName" value="entityManagerFactory"/>
</bean>
<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactory" />
</bean>
<tx:annotation-driven proxy-target-class="true" />
<bean id="persistenceExceptionTranslationPostProcessor"
class="org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor" />
<jpa:repositories base-package="my.package.repository" entity-manager-factory-ref="entityManagerFactory"/>
The FileUploadService snippet looks like this...
EntityManager batchEntityManager = entityManagerFactory.createEntityManager();
EntityTransaction transaction = batchEntityManager.getTransaction();
transaction.begin();
try (BufferedReader buffer = new BufferedReader(new InputStreamReader(file.getInputStream()))) {
buffer.lines()
.filter(StringUtils::isNotBlank)
.forEach(csvLine -> {
invoiceService.createInvoice(csvLine);
if (counter.incrementAndGet() % (updateFrequency.equals(0) ? 1 : updateFrequency) == 0) {
FileUpload fileUpload1 = fileUploadRepository.findOne(fileUpload.getId());
fileUpload1.setNumberOfSentRecords(sentCount.get());
fileUploadRepository.save(fileUpload1);
transaction.commit();
transaction.begin();
batchEntityManager.clear();
}
});
transaction.commit();
} catch (IOException ex) {
systemException.incrementAndGet();
log.error("Unexpected error while performing send task.", ex);
transaction.rollback();
}
// Update FileUpload status.
FileUpload fileUpload1 = fileUploadRepository.findOne(fileUpload.getId());
fileUpload1.setNumberOfSentRecords(sentCount.get());
if (systemException.get() != 0) {
fileUpload1.setStatus(FileUploadStatus.SYSTEM_ERROR);
} else {
fileUpload1.setStatus(FileUploadStatus.SENT);
}
fileUploadRepository.save(fileUpload1);
batchEntityManager.close();
Most of the DB queries were select statements that return the same results for each record being inserted. It was obvious that enabling EhCache as a Second-Level cache would have a significant performance improvement. However, this application has been running flawlessly in production for several years without ehcache enabled. I am hesitant to turn this on globally as I do not know how this will affect the large number of other repositories/queries.
Question 1
Is there a way to configure a "alternate" EntityManagerFactory that uses the second level cache for this batch process only? How can I choose to use this factory instead of the primary for this batch process only?
I experimented adding something like below to my spring config. I can easily inject this additional EntityManager into my class and use it. However, the existing Repositories (such as FileUploadRepository) don't seem to use it - they just return null. I am not sure if this approach is possible. The documentation for JpaTransactionManager says
This transaction manager is appropriate for applications that use a single JPA EntityManagerFactory for transactional data access
which is exactly what I am not doing. So what other options are there?
<bean id="batchEntityManagerFactory"
class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean" depends-on="flyway">
<property name="dataSource" ref="dataSource" />
<property name="packagesToScan" value="my.package.domain" />
<property name="loadTimeWeaver">
<bean class="org.springframework.instrument.classloading.InstrumentationLoadTimeWeaver" />
</property>
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter" />
</property>
<property name="jpaProperties">
<props>
<prop key="hibernate.hbm2ddl.auto">${hibernate.hbm2ddl.auto:validate}</prop>
<prop key="hibernate.dialect">org.hibernate.dialect.PostgreSQL9Dialect</prop>
<prop key="hibernate.show_sql">${hibernate.show_sql:false}</prop>
<prop key="hibernate.generate_statistics">${hibernate.generate_statistics:false}</prop>
<prop key="hibernate.ejb.interceptor">my.package.HibernateStatistics</prop>
<prop key="hibernate.cache.use_query_cache">true</prop>
<prop key="hibernate.cache.use_second_level_cache">true</prop>
<prop key="hibernate.cache.region.factory_class">org.hibernate.cache.ehcache.EhCacheRegionFactory</prop>
<prop key="hibernate.jdbc.batch_size">100</prop>
<prop key="hibernate.order_inserts">true</prop>
<prop key="hibernate.order_updates">true</prop>
</props>
</property>
<property name="persistenceUnitName" value="entityManagerFactory" />
</bean>
Question 2
Assuming there is no other option to "selectively" use EhCache, I tried enabling it on the primary EntityManagerFactory only. We can certainly do regression testing to make sure we don't introduce new issues. I assume this is fairly safe to do? However, another issue came up. I am trying to commit inserts in batches as described in this post and shown in my code above. I am getting an RollbackException when trying to commit the batch due to Connection org.postgresql.jdbc.PgConnection#1e7eb804 is closed.. I presume this is due to the maxWaitMillis property on the dataSource.
I don't want to change this property for every other existing spring Service/Repository/query in the application. If I could use the "custom" EntityManagerFactory I could easily provide a different DataSource Bean with the properties I want. Again, is this possible?
Maybe I looking at this problem all wrong. Are there any other suggestions?

You can have another EntityManagerFactory bean with a different qualifier, so that's one option. I would still recommend you look into these select queries. I bet the problem is just a missing index which causes the database to do full table scans. If you add the proper indexes, you probably don't need to change a thing in your application.

Manually open a hibernate session with Spring config

The issue I am having is that I use Spring to manage and load hibernate for my web application. I am currently using OpenSessionInViewFilter. This works as intended when I am viewing the application, but not so well when I am trying to access hibernate from non-view related activities such as a Quartz task or some Runnable thread I create to help with some tasks. This causes the Lazy initialize exception and no session available exceptions to occur.
Here is how I currently use Spring to manage Hibernate
<bean id="mainDataSource"
class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
[..DB config..]
</bean>
<bean id="sessionFactory"
class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
<property name="dataSource">
<ref bean="mainDataSource"/>
</property>
</bean>
<bean id="txManager"
class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory"><ref local="sessionFactory"/></property>
<property name="dataSource"><ref local="mainDataSource"/></property>
</bean>
I then configure DAO objects which extend HibernateDaoSupport and inject them into service classes
<bean id="myDAO"
class="package.myDAO">
<property name="sessionFactory">
<ref bean="sessionFactory" />
</property>
</bean>
<bean id="mySvcTarget" class="package.myService">
<property name="myDAO"><ref bean="myDAO"/></property>
</bean>
<bean id="myService"
class="org.springframework.transaction.interceptor.TransactionProxyFactoryBean">
<property name="transactionManager">
<ref bean="txManager"/>
</property>
<property name="target">
<ref bean="mySvcTarget"/>
</property>
<property name="transactionAttributes">
<props>
<prop key="*">PROPAGATION_REQUIRED</prop>
</props>
</property>
</bean>
So then in my application, myService is injected into my controller classes so I use that to get access to my DAO's. For my situation though it appears I need to access my DAO's (or service preferably) some other way and manually open and close my hibernate sessions since my service classes only seem to be open during view session. I am not exactly sure the best way to do this. All the hibernate configurations are there already in Spring so I'm assuming its just a matter or calling them somehow.

First of all those additional services that you're using (non-views) should be visible by Spring. The simplest way to do it is to use #Service annotation. And to make it work you can add <context:component-scan base-package="your.package"> in your configuration.
After this, if Spring sees your service as a bean, it should be enough to use #Transactional annotation to have Hibernate session in it.

Using HibernateTransactionManager for jdbc template

I was working on an spring application before which uses HibernateTransactionManager for transactions with queries/updates using jdbc template and works fine as well.
Below was the code used
<bean id="sybaseDataSource" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="MySybaseDS"/>
<property name="lookupOnStartup" value="false"/>
<property name="cache" value="true" />
<property name="proxyInterface" value="javax.sql.DataSource" />
</bean>
<bean id="sessionFactory" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
<property name="dataSource" ref="sybaseDataSource" />
<property name="mappingResources">
<list>..........</list>
</property>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">
org.hibernate.dialect.SybaseAnywhereDialect
</prop>
<prop key="hibernate.show_sql">true</prop>
</props>
</property>
</bean>
<tx:annotation-driven transaction-manager="txManager"/>
<bean id="txManager" class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory">
<ref local="sessionFactory"/>
</property>
</bean>
I am creating a new application and confused now whether to use the HibernateTransactionManager or JpaTransactionManager or both(possible?).
The application will use both HibernateTemplate and JdbcTemplate(for bulk updates) for database operations and need to be transactional.
I mean, can/should I use HibernateTransactionManager for transactions with jdbctemplate and HibernateTemplate database operations? Will there be any performance bottleneck of using one transaction manager over another?

The short answer is that it will work. See also this thread for a description of the low-level details.

If the application is JPA based (an entity manager factory is configured), then the JpaTransactionManager should be used.
This transaction manager binds an entity manager to the thread during the duration of the transaction.
The HibernateTransactionManager is for applications that don't use JPA but use instead use Hibernate directly (a session factory is configured).
This transaction manager binds a hibernate session to the thread instead of an entity manager.
They do basically the same, it's just one handles a Session while the other handles an EntityManager. Their choice is simply dictated by the fact that JPA is used or not.
Both transaction managers are compatible with the JdbcTemplate, according to their respective javadocs.
There are no performance consequences of using one transaction manager versus the other.
Internally the entity manager uses a Hibernate session, so the end result is the same: a hibernate session is binded to the thread directly or indirectly during the duration of the transaction.

Insert into database after spring hibernate sessionfactory is created

I want to insert some data in a table of database after the SessionFactory of spring hibernate has been created.
I have a CUSTOMER table and I want to insert a particular Customer into this table if the Customer doesn't exist.
This is the configuration of SessionFactory:
<bean id="sessionFactory" class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.MySQLDialect</prop>
<prop key="hibernate.show_sql">true</prop>
<prop key="hibernate.hbm2ddl.auto">update</prop>
</props>
</property>
<property name="packagesToScan" value="com.edfx.adb.persist.entity" />
<property name="entityInterceptor" ref="entityInterceptor" />
</bean>
I know that there is an option hibernate.hbm2ddl.import_files from which we can mention a sql file and the query will be executed at the time of SessionFactory initialization. But I don't want to follow this way.
Also I have found that we can use ServletContextListener for this kind of purpose, but I am not sure this is a good way to achieve what I want. Because what I understand is the listener will be initialized after deployment of war file, when the first user hit the application. So if I have a Scheduler which operates of the Customer and if the Scheduler get executed before any user hit the app, then the Scheduler wouldn't find that need-to-be-insert data.
Any pointer would be very helpful to me.

You can extend org.springframework.orm.hibernate4.LocalSessionFactoryBean and override buildSessionFactory() method to according to your requirement.

No Hibernate Session bound to thread, and configuration does not allow creation of non-transactional one here

What is this error about? "No Hibernate Session bound to thread, and configuration does not allow creation of non-transactional one here".
My spring config file looks something like this.
<bean id="jndiDataSource"
class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName">
<value>java:/devDS</value>
</property>
</bean>
<bean id="stsaDBFactory"
class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
<property name="dataSource" ref="jndiDataSource" />
<property name="annotatedClasses">
<list>
<value>xx.yy.zz.User</value>
<value>xx.yy.UserResponse</value>
</list>
</property>
<property name="hibernateProperties">
<props>
<prop key="hibernate.show_sql">true</prop>
<prop key="hibernate.hbmddl.auto">create</prop>
</props>
</property>
</bean>
<!-- ################################### Aspects ################################################## -->
<bean id="txManager"
class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory">
<ref local="stsaDBFactory" />
</property>
</bean>
All the DAO test passes when i test them outside of the container using junit. When I deploy it in jBoss as a portal app,I get this exception. Also it works fine if i remove the portal specific configuration and make it a simple web app and deploy it on jboss.Any idea?

You have defined a TransactionManager in your spring config but you are trying to execute a hibernate query in a method that is not transactional. Try adding #Transactional to your method or class.

I got around this problem by specifying the current_session_context_class in hibernate config to be "thread", as per the simple configuration shown in the hibernate configuration documentation.
But it recommends that its not safe for production usage.
Trying to add the following in your hibernate config should also help:
<property name="current_session_context_class">org.hibernate.context.ThreadLocalSessionContext</property>
Check out http://docs.jboss.org/hibernate/core/3.6/reference/en-US/html/architecture.html#architecture-current-session for more details.