Develop Reference

Debugging Memory leak - org.hibernate.engine.StatefulPersistenceContext

There is a service that connects to Oracle DB for reading data and it uses Hibernate-3.6 and SpringData-JPA-1.10.x. Heap dumps are getting generated frequently which results in out of memory on the hosts.
After analyzing few heapdumps using Eclipse MAT, found that the majority of the memory is accumulated in one instance of org.hibernate.engine.StatefulPersistenceContext -> org.hibernate.util.IdentityMap -> java.util.LinkedHashMap.
And the leak suspect says
The thread java.lang.Thread # 0x84427e10 ... : 29 keeps local
variables with total size 1,582,637,976 (95.04%) bytes.
The memory is accumulated in one instance of "java.util.LinkedHashMap"
loaded by "".
Searched it on StackOverflow and it says SessionFactory should be singleton and session.flush() and session.clear() should be invoked before each call to clear the cache. But SessionFactory is not explicitly initialized or used in the code.
What's causing the memory leak here (looks like the result of each query is cached and not cleared) and how to fix it?
More info about the Spring Data configuration:
TransactionManager is initialized as:
<tx:annotation-driven mode='proxy' proxy-target-class='true' />
<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactory" />
</bean>
<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
....
</bean>
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" depends-on="...">
....
</bean>
To interact with the table, an interface is declared extending Spring Data Repository and JpaSpecificationExecutor. Both are typed into the domain class that it will handle.
API activity method has the annotation #Transactional(propagation = Propagation.SUPPORTS, readOnly = true).

From what you describe this is what I expect to be going on:
Hibernate (actually JPA in general) keeps a reference to all entities it loads or saves for the lifetime of the session.
In a typical web application setup, this isn't a problem, because. A new session starts with each request and gets closed once the request is finished and it doesn't involve that many entities.
But for your application, it looks like the session keeps growing and growing.
I can imagine the following reasons:
something runs in an open session all the time without it ever closing. Maybe something like a batch job or a scheduled job which runs at regular intervals.
Hibernate is configured in such a way that it reuses the same session without ever closing it.
In order to find the culprit enable logging for opening and closing the session. Judging from https://hibernate.atlassian.net/browse/HHH-2425 org.hibernate.impl.SessionImpl should be the right log category and you probably need trace level logging.
Now test the various requests to your server and see if there are any sessions that get opened but not closed.

The question contains information about the creations of some beans. But the problem doesn't lie there. The problem is in your code, where have you use these beans.
Please check your code. Probably you are loading items in a loop. And the loop is wrapped with a transaction.
Hibernate creates huge intermediate objects, and it doesn't clean these before the transaction being completed (commit/rollback).

Classcastexception with spring ldap

We are changing our ldap from the java ldap library to spring's to take advantage of the rollback. It mostly works. Mostly.
One of our problems is that whenever an exception is thrown, instead of the exception bubbling out, we get this:
Caused by: java.lang.ClassCastException: org.springframework.ldap.transaction.compensating.manager.ContextSourceAndHibernateTransactionManager$ContextSourceAndHibernateTransactionObject incompatible with org.springframework.orm.hibernate3.HibernateTransactionManager$HibernateTransactionObject
at org.springframework.orm.hibernate3.HibernateTransactionManager.doSetRollbackOnly(HibernateTransactionManager.java:696) ~[spring-orm-3.2.0.RELEASE.jar:3.2.0.RELEASE]
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:852) ~[spring-tx-3.2.0.RELEASE.jar:3.2.0.RELEASE]
at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:822) ~[spring-tx-3.2.0.RELEASE.jar:3.2.0.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:410) ~[spring-tx-3.2.0.RELEASE.jar:3.2.0.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:114) ~[spring-tx-3.2.0.RELEASE.jar:3.2.0.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.2.0.RELEASE.jar:3.2.0.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) ~[spring-aop-3.2.0.RELEASE.jar:3.2.0.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.2.0.RELEASE.jar:3.2.0.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631) ~[spring-aop-3.2.0.RELEASE.jar:3.2.0.RELEASE]
This swallows up the exceptions. Yet, it doesn't seem to prevent the rollback from actually happening.
Relevant parts of our spring config:
<ldap:context-source
url="${LDAP_URL_SSL}"
base="${SEARCHBASE_DC}"
referral="follow"
authentication-source-ref="afmsAuthenticationSource" />
<ldap:ldap-template id="ldapTemplate" />
<bean id="txManager" name="txManager transactionManager" class="my.package.hibernate.HibernateTransactionManager">
<property name="sessionFactory" ref="enoteSessionFactory" />
<property name="defaultUserName" value="umc.not.provided" />
<property name="defaultApplicationId" value="umc" />
</bean>
<ldap:transaction-manager id="ldapTransactionManager" session-factory-ref="sessionFactory" >
<ldap:default-renaming-strategy />
</ldap:transaction-manager>
What you'll notice is that we have both a hibernate transaction manager (which we customized a long time ago and has worked forever) and the new contextsource transaction manager. Both are invoked via either AOP or #Transactional depending on which part of the code one is in.
We have a little disagreement in house as to whether we are supposed to REPLACE the existing tx manager with the ldap one (which might do both jobs) or ADD TO it. The dev doing the work thinks ADD TO. I think REPLACE, and i think this exception is a symptom of having both in play.
Anyway, can someone give advice on what might be causing this exception, and whether having one or both tx manager is the right way to use this library?

HibernateCursorItemReader Result set already closed

I'm using a spring batch HibernateCursorItemReader, it's defined as follows
<bean class="org.springframework.batch.item.database.HibernateCursorItemReader"
scope="step" id="priceListFctrItemReader">
<property name="queryName" value="FIND_ALL_PRICE_LIST_FCTR_ITEM_ID_BY_MONTRY_FCTR_VER"/>
<property name="sessionFactory" ref="sessionFactory"/>
<property name="parameterValues">
<map>
<entry key="factorVersion" value="#{jobParameters['current.factor.version']}"/>
<entry key="trueValue" value="#{true}"/>
</map>
</property>
</bean>
On small results it seems to be fine. But if processing takes long it seems the session closes and I get
org.hibernate.exception.GenericJDBCException: could not advance using next()
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:112)
and further down
Caused by: java.sql.SQLException: Result set already closed
at weblogic.jdbc.wrapper.ResultSet.checkResultSet(ResultSet.java:144)
at weblogic.jdbc.wrapper.ResultSet.preInvocationHandler(ResultSet.java:93)
I don't experience this in spring-boot, but on weblogic I do. It could be that local spring boot is just faster.
any ideas on how to avoid this ?

The problem is that spring-batch does a commit after every chunk, and committing closes the transaction and thus the result set.
When you are not in an application container, like when you are using spring boot, *CursorItemReaders use a separate connection to bypass the transaction and thereby avoid the commit which closes the cursor result set.
On the other hand, if you are running on an application server the connection you get from the server managed data source will by default take part in a transaction. In order for a cursor item reader to work you must set up a data source which does not take part in transactions.a
Alternatively, you may be able to use a *PagingItemReader which reads page size records per chunk, each in a separate transaction. This completely avoids the problem of a closing result set. Beware: If the underlying table changes between chunks the results may not be what you expect!
[a] : https://blog.codecentric.de/en/2012/03/transactions-in-spring-batch-part-2-restart-cursor-based-reading-and-listeners/

What's the right way to ensure jms consumers are closed using spring integration?

I'm using spring integration to invoke a service on the other end of an active mq. My config looks like:
<bean id="jmsConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<constructor-arg>
<bean class="org.apache.activemq.ActiveMQConnectionFactory"
p:brokerURL="${risk.approval.queue.broker}"
p:userName="${risk.approval.queue.username}"
p:password="${risk.approval.queue.password}"
/>
</constructor-arg>
<property name="reconnectOnException" value="true"/>
<property name="sessionCacheSize" value="100"/>
</bean>
<!-- create and close a connection to prepopulate the pool -->
<bean factory-bean="jmsConnectionFactory" factory-method="createConnection" class="javax.jms.Connection"
init-method="close" />
<integration:channel id="riskApprovalRequestChannel"/>
<integration:channel id="riskApprovalResponseChannel"/>
<jms:outbound-gateway id="riskApprovalServiceGateway"
request-destination-name="${risk.approval.queue.request}"
reply-destination-name="${risk.approval.queue.response}"
request-channel="riskApprovalRequestChannel"
reply-channel="riskApprovalResponseChannel"
connection-factory="jmsConnectionFactory"
receive-timeout="5000"/>
<integration:gateway id="riskApprovalService" service-interface="com.my.super.ServiceInterface"
default-request-channel="riskApprovalRequestChannel"
default-reply-channel="riskApprovalResponseChannel"/>
What I've noticed is that with this config the consumers created to grab the matching request from active mq never close. Every request increments the consumer count.
I can stop this from happening by adding
<property name="cacheConsumers" value="false" />
To the CachingConnectionFactory.
However according to the java docs for CachingConnectionFactory :
Note that durable subscribers will only be cached until logical
closing of the Session handle.
Which suggests that the session is never being closed.
Is this a bad thing? Is there a better way to stop the consumers from piling up?
Cheers,
Peter

First, you don't need the init-method on your factory-bean - it does nothing - the session factory only has one connection and calling close() on it is a no-op. (CCF is a subclass of SingleConnectionFactory).
Second; caching consumers is the default; sessions are never closed, unless the number of sessions exceeds the sessionCacheSize (which you have set to 100).
When close() is called on a cached session, it is cached for reuse; that's what the caching connection factory is for - avoiding the overhead of session creation for every request.
If you don't want the performance benefit of caching sessions, producers and consumers, use the SingleConnectionFactory instead. See the JavaDoc for CachingConnectionFactory.

Does the following work when using cachingConnectionFactory?
In your spring config file add in the connection factory config details something like this: cacheConsumers="false"
Default Behaviour is true which was causing a connection leak in the Queue.

Concurrency issues in Spring DAOs with 3.0.0.RC1

After upgrading from Spring 3.0.0.M4 to 3.0.0.RC1 and Spring Security 3.0.0.M2 to 3.0.0.RC1, I've had to use a security:authentication-manager tag instead of defining an _authenticationManager like I used to in M4/M2. I've done my best at defining it, and ended up with this:
<security:authentication-manager alias="authenticationManager">
<security:authentication-provider user-service-ref="userService">
<security:password-encoder hash="plaintext"/>
</security:authentication-provider>
</security:authentication-manager>
When I do my unit tests one at a time, this works great, and for most AJAX requests it works fine as well, but seemingly randomly, I get weird errors in my transactions where my database session seems to get closed midway in the work. The way I can provoke these errors is just sending a lot of different AJAX requests to my different controllers from the same client, then at least one of them will fail at random. Next time I try, that one will work and another will fail.
The error happens most frequently in my userDAO, but also quite frequently in other DAOS, and the exceptions include at least the following:
"java.sql.SQLException: Operation not allowed after ResultSet closed"
"org.hibernate.impl.AbstractSessionImpl:errorIfClosed(): Session is closed!"
"java.lang.NullPointerException at com.mysql.jdbc.PreparedStatement.fillSendPacket(PreparedStatement.java:2439)"
"java.util.ConcurrentModificationException at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(Unknown Source)"
"org.hibernate.LazyInitializationException: illegal access to loading collection"
etc...
Before, I used to define an _authenticationManager bean, and the same requests worked like a charm. But with RC1, I'm no longer allowed to define it. It used to look like this:
<bean id="_authenticationManager" class="org.springframework.security.authentication.ProviderManager">
<property name="providers">
<list>
<bean class="org.springframework.security.authentication.dao.DaoAuthenticationProvider">
<property name="userDetailsService" ref="userService"/>
<property name="passwordEncoder">
<bean class="org.springframework.security.authentication.encoding.PlaintextPasswordEncoder" />
</property>
</bean>
</list>
</property>
</bean>
Have I defined my security:authentication-manager incorrectly so that it will share transactions for multiple requests from the same client? Should I define it differently, or should I define some other security: beans?
Is there something I have misunderstood that makes my database sessions close? In my head, each request has its own database connection and transaction. All getters and setters are synchronized methods, so I really shouldn't have any concurrency issues. All the REST controllers that the UI makes requests against are GET-requests and do read-only work. To my knowledge, not a single INSERT/UPDATE/DELETE is done during any of these requests, and I've inspected the database logs to verify this.
I look forward to hearing your suggestions on how to avoid these race-conditions.
Cheers
Nik
PS, my I've updated the question to be more specific that the problem is with the security:authentication-manager (or so it seems to me, if you have tips that it could be something else that would be great) that I'm forced to use instead of my own _authenticationManager starting with 3.0.0.RC1
PPS, here is the thread that made me understand I could no longer define an _authenticationManager: SpringSource Forum Post

It seems that I had a big problem in database session handling in my DAO, so I've made a write-up of my problem and posted the solution in another thread here at StackOverflow and asked for people's opinion on the solution. I hope it doesn't give more issues :-)
Cheers
Nik