I have a Spring Boot RESTful CRUD service in car rental domain.
From the high overview, it's a simple CRUD app with SQL database and all such entities as a Car, Client, Lease, etc.
Now I have to introduce a report generation feature that aimed to process lease data and calculate some statistics based on data in SQL db and persist the report into MongoDB.
I've already implemented it by creating a ReportGenerationService that depends on OriginDataService and MongoService.
ReportGenerationService generates the report based on a data returned by OriginDataService. In turn, OriginDataService has a method getData() that do a number of calls to DAO layer and thus annotated with #Transactional(isolation = Isolation.REPEATABLE_READ). I want the returned data to be consistent. After getting the data ReportGenerationService generates a report and persists it by invoking MongoService's persist(Report) method.
In my implementation I get data -> generate report -> persist report.
But what if the base data and report can't fit into RAM?
The solution is to select it little by little, generate a part of the report, persist the part of the report and after all rows of data is processed merge the report.
It means that one method should read the data, processes it and persists.
I also want my method to read data with Repeatable Read isolation level, then I have to annotate the method with #Transactional(isolation = Isolation.REPEATABLE_READ). But since 2 dbs are used in the method the #Transactional will spread on both of them, and I want only SQL to use it.
How can I do gradually reads and writes to different dbs?
Refer below links and code sample, it may help you to resolve your issue.
https://www.javaworld.com/article/2077963/distributed-transactions-in-spring--with-and-without-xa.html?page=2
Transaction management for multiple database Using Spring & Hibernate
<bean id="transactionManager" class="com.springsource.open.db.ChainedTransactionManager">
<property name="transactionManagers">
<list>
<bean
class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource" />
</bean>
<bean
class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="otherDataSource" />
</bean>
</list>
</property>
</bean>
Related
There is a service that connects to Oracle DB for reading data and it uses Hibernate-3.6 and SpringData-JPA-1.10.x. Heap dumps are getting generated frequently which results in out of memory on the hosts.
After analyzing few heapdumps using Eclipse MAT, found that the majority of the memory is accumulated in one instance of org.hibernate.engine.StatefulPersistenceContext -> org.hibernate.util.IdentityMap -> java.util.LinkedHashMap.
And the leak suspect says
The thread java.lang.Thread # 0x84427e10 ... : 29 keeps local
variables with total size 1,582,637,976 (95.04%) bytes.
The memory is accumulated in one instance of "java.util.LinkedHashMap"
loaded by "".
Searched it on StackOverflow and it says SessionFactory should be singleton and session.flush() and session.clear() should be invoked before each call to clear the cache. But SessionFactory is not explicitly initialized or used in the code.
What's causing the memory leak here (looks like the result of each query is cached and not cleared) and how to fix it?
More info about the Spring Data configuration:
TransactionManager is initialized as:
<tx:annotation-driven mode='proxy' proxy-target-class='true' />
<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactory" />
</bean>
<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
....
</bean>
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" depends-on="...">
....
</bean>
To interact with the table, an interface is declared extending Spring Data Repository and JpaSpecificationExecutor. Both are typed into the domain class that it will handle.
API activity method has the annotation #Transactional(propagation = Propagation.SUPPORTS, readOnly = true).
From what you describe this is what I expect to be going on:
Hibernate (actually JPA in general) keeps a reference to all entities it loads or saves for the lifetime of the session.
In a typical web application setup, this isn't a problem, because. A new session starts with each request and gets closed once the request is finished and it doesn't involve that many entities.
But for your application, it looks like the session keeps growing and growing.
I can imagine the following reasons:
something runs in an open session all the time without it ever closing. Maybe something like a batch job or a scheduled job which runs at regular intervals.
Hibernate is configured in such a way that it reuses the same session without ever closing it.
In order to find the culprit enable logging for opening and closing the session. Judging from https://hibernate.atlassian.net/browse/HHH-2425 org.hibernate.impl.SessionImpl should be the right log category and you probably need trace level logging.
Now test the various requests to your server and see if there are any sessions that get opened but not closed.
The question contains information about the creations of some beans. But the problem doesn't lie there. The problem is in your code, where have you use these beans.
Please check your code. Probably you are loading items in a loop. And the loop is wrapped with a transaction.
Hibernate creates huge intermediate objects, and it doesn't clean these before the transaction being completed (commit/rollback).
I am using Spring + Hibernate on my JavaEE project.
In this project the user can upload an XLS file which I should import to my database. Before importing I have to validate this file checking its integrity with the other entities on my database. So I have more or less the following:
// The importer
#Component("importer")
public class Importer {
#Autowired
FirstDAO firstDao;
#Autowired
SecondDAO secondDao;
// Read the file and open it (65.000 lines for example)
public void validate() {
foreach line in the file {
firstDAO.has(line[col1]);
secondDao.has(line[col2]);
}
// It stores the valid objects in a List and persist them at the end
}
}
// The DAO
#Repository
public class FirstDao {
#PersistenceContext
protected EntityManager entityManager;
#Transactional(propagation = Propagation.NOT_SUPPORTED)
public boolean has(String name) {
List<Object> result = entityManager.createQuery( from FIRST_TABLE where name = :name)
.setParameter("name", name)
.getResultList();
if (result.size > 0) return true;
else return false;
}
}
// The PersistenceContext/Hibernate configuration
<!-- Data Source -->
<jee:jndi-lookup id="myDS" jndi-name="jdbc/my-DS" cache="true" proxy-interface="javax.sql.DataSource" />
<!-- Entity Manager -->
<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property value="classpath:META-INF/my_persistence.xml" name="persistenceXmlLocation"/>
<property name="dataSource" ref="myDS"/>
<property name="persistenceUnitName" value="myPersistenceUnit" />
<!--
<property name="loadTimeWeaver">
<bean class="org.springframework.instrument.classloading.InstrumentationLoadTimeWeaver"/>
</property>
-->
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="database" value="ORACLE" />
<property name="showSql" value="false" />
</bean>
</property>
</bean>
After logging the application I have noticed:
For each query (has method on my DAO) a connection is opened and closed with my Database.
The memory on the server is being flooded (probably memory leak).
After a lot of opening and closing connections I have a connection reset from the Database. Don't know why. And if I still keep requesting coonections, the Datasource is suspended.
I have read somethings about entityManager but I still don't know if I am doing it right, so:
Is it right to execute the validation in a for loop that way? (One connection for each item, meaning 130.000 connections open and closed in a 65000 lines file)
I have read about Stateless Persistence Context for the entityManager. I suspect the memory leak may be there. Maybe Hibernate is kepting a lot of objects in the PersistenceContext. How do I tell Entity Manager to not cache those guys when validating?
Thanks in advance.
First of all, you really shouldn't do that line by line unless you have a very very good reason. Even if the data size is bigger than your memory you should do that 1000 lines at a time or something like that but definitely not one by one.
Because one of the most important optimization for database usage is reducing number of database hit.
Secondly you should not retrieve the data just to check if it is exist.
You should use a basic "select count" query. By that way you will get rid of all stuff like consuming IO to read data and retrieving that data through network to your server and spending memory to just get the number of object in that list.
If you will use my first advice and check the existing of records not one at a time but 1000s at a time you can select just the names instead of all rows.
Btw as far as I can see you are using a datasource if that is properly configured like number of max connection etc. you shouldn't worry about number of database connection.
I am working on spring batch, for writer currently using FlatFileItemWriter.
I would like to write my input file content to more than one flat file based on some field value. Is Spring batch support any kind of functionality by default.[something similar to CompositeItemWriter]
For example, my input file content is something like this.
john,35,retail,10000
joe,34,homeloan,20000
Amy,23,retail,2000
Now i would like to write two different files based on third column, it means row 1 and row 3 should go to file1 and row 2 should go to file2.
My writer configuration is:
<bean id="fileWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" value="file:C:/output.dat"/>
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter" value="|" />
<property name="fieldExtractor">
<bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names" value="field1,field2...." />
</bean>
</property>
</bean>
</property>
</bean>
Take a look at the ClassifierCompositeItemWriter. This ItemWriter implementation allows you to define a Classifier that chooses which of the defined delegate ItemWriter instances to delegate to. In your case, you'd create a Classifier that decided based on field4 and delegate the writing to the appropriate instance of the FlatFileItemWriter.
You can read more about the ClassifierCompositeItemWriter in the documentation here: http://docs.spring.io/spring-batch/trunk/apidocs/org/springframework/batch/item/support/ClassifierCompositeItemWriter.html
Use a ClassifierCompositeItemWriter
Calls one of a collection of ItemWriters for each item, based on a
router pattern implemented through the provided Classifier.
Router pattern is based on bean content
I am having an issue in saving and retrieving objects in database in just one request.
I want to clear the cache of our hibernate session to get the updated entity in our database.
My code looks like this:
public class SampleController{
protected ModelAndView onSubmit(HttpServletRequest request, HttpServletResponse response, Object command, BindException errors)
throws Exception {
myServiceOne.doAllotsOfSaving(parameters);
//some code enhancements to remove cache in hibernate session
//without affecting the session of other user logged in.
//some fields in MyEntity class contains the old values but the actual data in database is already updated
MyEntity entity = myServiceTwo.getMyEntityByOrderNo(orderNo);
}
}
--configurations
<bean id="sessionFactory" class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="configLocation" value="classpath:hibernate.cfg.xml" />
<property name="hibernateProperties">
<ref local="hibernateProperties"/>
</property>
<property name="entityInterceptor">
<ref bean="auditLogInterceptor" />
</property>
</bean>
<bean id="myServiceOne" class="com.test.service.impl.MyServiceOneImpl">
<property name="sessionFactory" ref="sessionFactory" />
</bean>
<bean id="myServiceTwo" class="com.test.service.impl.MyServiceTwoImpl">
<property name="sessionFactory" ref="sessionFactory" />
</bean>
--configurations
Does java web application contains only one hibernate session and how to clear this hibernate session?
No any hibernate based application must use multiple sessions. Each of these sessions must be closed when they perform their task. Hibernate can manage sessions for you if you configure the same in hibernate's configuration file.
However you should have only one instance of SessionFactory per application.
To clear the session you can call the session.clear() method. It clears the session level cache.
without affecting the session of other user logged in
Since you have a web application, you have a different thread for each user for database transactions. This means each user will have a different hibernate session, so you won't have to worry about this. If by some means you're using same session for all the users, you're doing it wrong and results can be catastrophic. After some time you'll get an OutOfMemoryError because of session level cache.
You must note that you cannot disable hibernate session level cache. For this purpose you may use StatelessSession.
Session factory is long live multithreaded object.
Usually one session factory should be created for one database.
A Session is used to get a physical connection with a database. The Session object is lightweight and designed to be instantiated each time an interaction is needed with the database.
The main function of the Session is to offer CRUD operations for instances of mapped entity classes. Instances may exist in one of the following three states at a given point in time:
transient: A new instance of a a persistent class which is not associated with a Session and has no representation in the database and no identifier value is considered transient by Hibernate.
persistent: You can make a transient instance persistent by associating it with a Session. A persistent instance has a representation in the database, an identifier value and is associated with a Session.
detached: Once we close the Hibernate Session, the persistent instance will become a detached instance.
I understand that Hibernate's Session.flush() method writes current data state in memory into the database. I use it and it works fine. But my question is: using Session.flush() is mandatory?
If I remove Session.flush() then nothing gets inserted/updated in the database. And I don't see any errors in my log file.
I am using Spring + Hibernate in my application. I am also using the OpenSessionInViewFilter which I have defined in my web.xml as follows:
<filter>
<filter-name>hibernateFilter</filter-name>
<filter-class>org.springframework.orm.hibernate3.support.OpenSessionInViewFilter</filter-class>
<init-param>
<param-name>flushMode</param-name>
<param-value>AUTO</param-value>
</init-param>
</filter>
<filter-mapping>
<filter-name>hibernateFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
I have tried using flushMode as COMMIT but still it didn't help.
My Spring applicationContext.xml has the following lines for datasource:
<bean id="dataSource" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="${jdbc.jndiName}" />
<property name="resourceRef" value="true" />
</bean>
<bean id="sessionFactory" class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
<property name="dataSource" ref="dataSource"></property>
<property name="hibernateProperties">
<props>
<prop key="hibernate.show_sql">${hibernate.show_sql}</prop>
<prop key="hibernate.dialect">${hibernate.dialect}</prop>
</props>
</property>
<property name="annotatedClasses">
<list>
<value>...</value>
</list>
</property>
</bean>
And finally my code snippet looks like this:
public String saveRegistration (final Registration registration) {
getHibernateTemplate().saveOrUpdate(registration);
getSession().flush(); // If I remove this, no records would be inserted/updated!!
String id = registration.getId();
return id;
}
As you can see I am using a very basic configurations in Spring and Hibernate.
Can someone please help me in understanding as to why getSession.flush() should be called? Why without it no records are getting saved/updated in the database?
Are you certain that no records get created in the database? If you are saying that because the returned id is null then I think you've misunderstood. When you call saveOrUpdate, the object will be stored in the Hibernate session and other code that runs a query can retrieve it, but it won't necessarily be persisted to the database immediately. The current object instance won't have an id until you run a query that prompts Hibernate to persist the object to the database, or you manually call flush().
I'd suggest you check what the behaviour is if you code a second method to query for the object. I think you'll find that causes Hibernate to flush the object to the database.
Probably parameter "registration" is not been changed once it's loaded from database or saved. Hibernate will save/update entity when there is some difference between cached object and database data.
Try after modifying one of the field in "registration" instance.
Also, it would be good idea to turn on Hibernate/Spring debug logging and see what's going on under the hood.
The flush is just synchronization with the db and then you need to commit or clear the session to see the changes.
And if you do just commit .It automatically flushes and you can see your changes from another session too.
Commit will make the database commit
Flushing is the process of synchronizing the underlying persistent
store with persistable state held in memory.
ie. it will update or insert into your tables in the running
transaction, but it may not commit those changes (this depends on
your flush mode).