Develop Reference

Spring #Transactional not working when Hibernate object with lazy loading coming from another transaction is passed down to method

I have a problem with accessing data inside a running transaction when the data came from another (supposedly closed) transaction. I have three classes like below, with an entity (called MyEntity) which also has another entity connected via Hibernate mapping called "OtherEntity" which has lazy loading set to true. Notice how I have two transactions:
One to load a list of entities
And a new transaction for each new item
However, this fails inside the loop with "No session" even though I have an active transaction inside the method (TransactionSynchronizationManager.isActualTransactionActive is true).
I don't really understand the problem. Seems to me the object which is used by the second transaction(s) "belong" to the first one even though the first transaction was supposed to finish? Maybe its a race condition?
#Service
class ServiceA {
#Autowired
private ServiceB serviceB;
#Autowired
private ServiceC serviceC;
public void test() {
List<MyEntity> allEntities = serviceC.loadAllEntities(); //First transaction ran, getting a list of entities, but due to lazy loading we havent loaded all the data
for(MyEntity i : allEntities) {
serviceB.doOnEach(i); //On each element a new transaction should start
}
}
}
#Service
class ServiceB {
#Transactional
public void doOnEach(MyEntity entity) {
System.out.println(TransactionSynchronizationManager.isActualTransactionActive()); //true, therefore we have an active transaction here
OtherEntity other = entity.getSomeOtherEntity(); //Want to load the "lazy loaded" entity here
//"No Session" exception is thrown here
}
}
#Service
class ServiceC {
#Autowired
private MyRepository myRepository;
#Transactional
public List<MyEntity> loadAllEntities() {
return myRepository.findAll();
}
}
A solution would be to re-load the "MyEntity" instance inside the "doOnEach" method, but that seems to me like a sub-optimal solution, especially on big lists. Why would I reload all the data which is already supposed to be there?
Any help is appreciated.
Obviously the real code is a lot more complicated than this but I have to have these kind of separate transactions for business reasons, so please no "solutions" which re-write the core logic of this. I just want to understand whats going on here.

After the call to loadAllEntities() finishes the Spring proxy commits the transaction and closes the associated Hibernate Session. This means you cannot have Hibernate transparently load the non-loaded lazy associations anymore without explicitly telling it to do so.
If for some reason you really want your associated entities to be loaded lazily the two options you have is either use Hibernate.initialize(entity.getSomeOtherEntity()) in your doOnEach() method or set the spring.jpa.open-in-view property to true to have the OpenSessionInViewInterceptor do it for you.
Otherwise it's a good idea to load them together with the parent entity either via JOIN FETCH in your repository query or via an Entity Graph.
References:
https://www.baeldung.com/spring-open-session-in-view
https://www.baeldung.com/hibernate-initialize-proxy-exception
To clarify further:
Spring creates a transaction and opens a new Session (A) before entering the loadAllEntities() method and commits/closes them upon returning. When you call entity.getSomeOtherEntity() the original Session (A) that loaded entity is gone (i.e. entity is detached) but instead there's a new Session (B) which was created upon entering the doOnEach() transactional method. Obviously Session (B) doesn't know anything about entity and its relations and at the same time the Hibernate proxy of someOtherEntity inside entity references the original Session (A) and doesn't know anything about Session (B). To make the Hibernate proxy of someOtherEntity actually use the current active Session (B) you can call Hibernate.initialize().

#Transactional annotation Spring boot 2.0 and hibernate LazyInitializationException

I have the following question. From what I understand the #Transactional annotation is supposed to keep the session alive, thus enabling to lazy fetch child entities without the need to performe a specific joining query.
I have the following scenario where I do not understand why I'm still getting a LazyInitializationException.
My app runs a resolver in order to provide the various controller services with a resolved object so that it can be used directly.
Said resolver intercepts a header from the request and using it's value attempts to query the db in order to fetch the object. Now the object in question is quite simple is it's doings albeit it has a list of two sub-entities.
In order to perform the resolving action I'm using an extra service where I basically wrap some JpaRepository methods. The complete is below:
#Service
public class AppClientServiceImpl implements AppClientService {
private static final Logger LOGGER = LoggerFactory.getLogger(AppClientServiceImpl.class.getCanonicalName());
private final AppClientRepository repository;
#Autowired
public AppClientServiceImpl(AppClientRepository repository) {
this.repository = repository;
}
#Override
#Transactional(readOnly = true)
public AppClient getByAppClientId(final String appClientId) {
LOGGER.debug("Attempting to retrieve appClient with id:: {}", appClientId);
return repository.findByAppClientId(appClientId);
}
#Override
#Transactional
public void saveAndFlush(final AppClient appClient) {
LOGGER.debug("Attempting to save/update appClient:: {}", appClient);
repository.saveAndFlush(appClient);
}
}
As you can see both methods are annotated as #Transactional meaning that the should keep the session alive in the context of that said method.
Now, my main questions are the following:
1) Using the debugger I'm seeing even on that level getByAppClientId the list containing on the sub-entities which is lazy loaded has been resolved just fine.
2) On the resolver itself, where the object has been received from the delegating method, the list fails to be evaluated due to a LazyInitializationException.
3) Finally on the final controller service method which is also marked as #Transactional, the same as above occurs meaning that this eventually fails to it's job (since it's performing a get of the list that has failed to initialize.
Based on all the above, I would like to know what is the best approach in handling this. For once I do not want to use an Eager fetching type and I would also like to avoid using fetch queries. Also marking my resolver as #Transactional thus keeping the session open there as well is also out of the question.
I though that since the #Transactional would keep the session open, thus enabling the final service method to obtain the list of sub-entities. This seems not to be the case.
Based on all the above it seems that I need a way for the final service method that gets call (which needs the list on hand) to fetch it somehow.
What would the best approach to handle this? I've read quite a few posts here, but I cannot make out which is the most accepted methods as of Spring boot 2.0 and hibernate 5.
Update:
Seems that annotating the sub-entitie with the following:
#Fetch(FetchMode.SELECT)
#LazyCollection(LazyCollectionOption.TRUE)
Resolves the problem but I still don't know whether this is the best approach.

You initialize the collection by debugging. The debugger usually represents collections in a special way by using the collection methods which trigger the initialization, so that might be the reason why it seems to work fine during debugging. I suppose the resolver runs outside of the scope of the getByAppClientId? At that point the session is closed which is why you see the exception.
I created Blaze-Persistence Entity Views for exactly that use case. You essentially define DTOs for JPA entities as interfaces and apply them on a query. It supports mapping nested DTOs, collection etc., essentially everything you'd expect and on top of that, it will improve your query performance as it will generate queries fetching just the data that you actually require for the DTOs.
The entity views for your example could look like this
#EntityView(AppClient.class)
interface AppClientDto {
String getName();
}
Querying could look like this
List<AppClientDto> dtos = entityViewManager.applySetting(
EntityViewSetting.create(AppClientDto.class),
criteriaBuilderFactory.create(em, AppClient.class)
).getResultList();

How to refresh JPA entities when backend database changes asynchronously?

I have a PostgreSQL 8.4 database with some tables and views which are essentially joins on some of the tables. I used NetBeans 7.2 (as described here) to create REST based services derived from those views and tables and deployed those to a Glassfish 3.1.2.2 server.
There is another process which asynchronously updates contents in some of tables used to build the views. I can directly query the views and tables and see these changes have occured correctly. However, when pulled from the REST based services, the values are not the same as those in the database. I am assuming this is because JPA has cached local copies of the database contents on the Glassfish server and JPA needs to refresh the associated entities.
I have tried adding a couple of methods to the AbstractFacade class NetBeans generates:
public abstract class AbstractFacade<T> {
private Class<T> entityClass;
private String entityName;
private static boolean _refresh = true;
public static void refresh() { _refresh = true; }
public AbstractFacade(Class<T> entityClass) {
this.entityClass = entityClass;
this.entityName = entityClass.getSimpleName();
}
private void doRefresh() {
if (_refresh) {
EntityManager em = getEntityManager();
em.flush();
for (EntityType<?> entity : em.getMetamodel().getEntities()) {
if (entity.getName().contains(entityName)) {
try {
em.refresh(entity);
// log success
}
catch (IllegalArgumentException e) {
// log failure ... typically complains entity is not managed
}
}
}
_refresh = false;
}
}
...
}
I then call doRefresh() from each of the find methods NetBeans generates. What normally happens is the IllegalArgumentsException is thrown stating somethng like Can not refresh not managed object: EntityTypeImpl#28524907:MyView [ javaType: class org.my.rest.MyView descriptor: RelationalDescriptor(org.my.rest.MyView --> [DatabaseTable(my_view)]), mappings: 12].
So I'm looking for some suggestions on how to correctly refresh the entities associated with the views so it is up to date.
UPDATE: Turns out my understanding of the underlying problem was not correct. It is somewhat related to another question I posted earlier, namely the view had no single field which could be used as a unique identifier. NetBeans required I select an ID field, so I just chose one part of what should have been a multi-part key. This exhibited the behavior that all records with a particular ID field were identical, even though the database had records with the same ID field but the rest of it was different. JPA didn't go any further than looking at what I told it was the unique identifier and simply pulled the first record it found.
I resolved this by adding a unique identifier field (never was able to get the multipart key to work properly).

I recommend adding an #Startup #Singleton class that establishes a JDBC connection to the PostgreSQL database and uses LISTEN and NOTIFY to handle cache invalidation.
Update: Here's another interesting approach, using pgq and a collection of workers for invalidation.
Invalidation signalling
Add a trigger on the table that's being updated that sends a NOTIFY whenever an entity is updated. On PostgreSQL 9.0 and above this NOTIFY can contain a payload, usually a row ID, so you don't have to invalidate your entire cache, just the entity that has changed. On older versions where a payload isn't supported you can either add the invalidated entries to a timestamped log table that your helper class queries when it gets a NOTIFY, or just invalidate the whole cache.
Your helper class now LISTENs on the NOTIFY events the trigger sends. When it gets a NOTIFY event, it can invalidate individual cache entries (see below), or flush the entire cache. You can listen for notifications from the database with PgJDBC's listen/notify support. You will need to unwrap any connection pooler managed java.sql.Connection to get to the underlying PostgreSQL implementation so you can cast it to org.postgresql.PGConnection and call getNotifications() on it.
An an alternative to LISTEN and NOTIFY, you could poll a change log table on a timer, and have a trigger on the problem table append changed row IDs and change timestamps to the change log table. This approach will be portable except for the need for a different trigger for each DB type, but it's inefficient and less timely. It'll require frequent inefficient polling, and still have a time delay that the listen/notify approach does not. In PostgreSQL you can use an UNLOGGED table to reduce the costs of this approach a little bit.
Cache levels
EclipseLink/JPA has a couple of levels of caching.
The 1st level cache is at the EntityManager level. If an entity is attached to an EntityManager by persist(...), merge(...), find(...), etc, then the EntityManager is required to return the same instance of that entity when it is accessed again within the same session, whether or not your application still has references to it. This attached instance won't be up-to-date if your database contents have since changed.
The 2nd level cache, which is optional, is at the EntityManagerFactory level and is a more traditional cache. It isn't clear whether you have the 2nd level cache enabled. Check your EclipseLink logs and your persistence.xml. You can get access to the 2nd level cache with EntityManagerFactory.getCache(); see Cache.
#thedayofcondor showed how to flush the 2nd level cache with:
em.getEntityManagerFactory().getCache().evictAll();
but you can also evict individual objects with the evict(java.lang.Class cls, java.lang.Object primaryKey) call:
em.getEntityManagerFactory().getCache().evict(theClass, thePrimaryKey);
which you can use from your #Startup #Singleton NOTIFY listener to invalidate only those entries that have changed.
The 1st level cache isn't so easy, because it's part of your application logic. You'll want to learn about how the EntityManager, attached and detached entities, etc work. One option is to always use detached entities for the table in question, where you use a new EntityManager whenever you fetch the entity. This question:
Invalidating JPA EntityManager session
has a useful discussion of handling invalidation of the entity manager's cache. However, it's unlikely that an EntityManager cache is your problem, because a RESTful web service is usually implemented using short EntityManager sessions. This is only likely to be an issue if you're using extended persistence contexts, or if you're creating and managing your own EntityManager sessions rather than using container-managed persistence.

You can either disable caching entirely (see: http://wiki.eclipse.org/EclipseLink/FAQ/How_to_disable_the_shared_cache%3F ) but be preparedto a fairly large performance loss.
Otherwise, you can perform a clear cache programmatically with
em.getEntityManagerFactory().getCache().evictAll();
You can map it to a servlet so you can call it externally - this is better if your database is modify externally very seldom and you just want to be sure JPS will pick up the new version

Just a thought, but how do you receive your EntityManager/Session/whatever?
If you queried the entity in one session, it will be detached in the next one and you will have to merge it back into the persistence context to get it managed again.
Trying to work with detached entities may result in those not-managed exceptions, you should re-query the entity or you could try it with merge (or similar methods).

JPA doesn't do any caching by default. You have to explicitly configure it. I believe its the side effect of the architectural style you have chosen: REST. I think caching is happening at the web servers, proxy servers etc. I suggest you read this and debug more.

How to test whether lazy loaded JPA collection is initialized?

I have a service that gets a JPA entity from outside code. In this service I would like to iterate over a lazily loaded collection that is an attribute of this entity to see if the client has added something to it relative to the current version in the DB.
However, the client may have never touched the collection so it's still not initialized. This results in the well known
org.hibernate.LazyInitializationException: failed to lazily initialize a collection of role: com.example.SomeEntity.
Of course, if the client never touched the collection, my service doesn't have to check it for possible changes. The thing is that I can't seem to find a way to test whether the collection is initialized or not. I guess I could call size() on it and if it throws LazyInitializationException I would know, but I'm trying not to depend on such patterns.
Is there some isInitialized() method somewhere?

Are you using JPA2?
PersistenceUnitUtil has two methods that can be used to determine the load state of an entity.
e.g. there is a bidirectional OneToMany/ManyToOne relationship between Organization and User.
public void test() {
EntityManager em = entityManagerFactory.createEntityManager();
PersistenceUnitUtil unitUtil =
em.getEntityManagerFactory().getPersistenceUnitUtil();
em.getTransaction().begin();
Organization org = em.find(Organization.class, 1);
em.getTransaction().commit();
Assert.assertTrue(unitUtil.isLoaded(org));
// users is a field (Set of User) defined in Organization entity
Assert.assertFalse(unitUtil.isLoaded(org, "users"));
initializeCollection(org.getUsers());
Assert.assertTrue(unitUtil.isLoaded(org, "users"));
for(User user : org.getUsers()) {
Assert.assertTrue(unitUtil.isLoaded(user));
Assert.assertTrue(unitUtil.isLoaded(user.getOrganization()));
}
}
private void initializeCollection(Collection<?> collection) {
// works with Hibernate EM 3.6.1-SNAPSHOT
if(collection == null) {
return;
}
collection.iterator().hasNext();
}

org.hibernate.Hibernate.isInitialized(..)
There is no standard JPA solution to my knowledge. But if you want to actually initialize collections, you can create an utility method and iterate them (only one iteration is enough).

For eclipselink, users cast the collection you are trying to access to an org.eclipse.persistence.indirection.IndirectList, and then call its isInstantiated() method. The following link has more information:
http://www.eclipse.org/eclipselink/api/1.1/org/eclipse/persistence/indirection/IndirectList.html#isInstantiated.

Grails: eager loading doesn't work

I want to access the list of organisations from a user object within the main.gsp:
<g:select name="effectiveOrganisation"
from="${session.user.organisations}" optionKey="id" optionValue="name"
value="${session.effectiveOrganisation?.id}" />
The user object is defined by the following class:
class SystemUser {
static hasMany = [organisations: Organisation]
static belongsTo = [Organisation]
static mapping = {
organisations lazy: false
}
}
But when I execute my code, I get:
Exception Message: could not initialize proxy - no Session
Caused by: Error executing tag <g:form>:
Error executing tag <g:select>: could not initialize proxy - no Session
Why does the eager not work here?

It's not clear from your code, but I'm going to assume that you have a many-to-many here based on the belongsTo property.
I've managed to reproduce this with Grails 1.3.5. The problem only seems to affect the side of the relationship that has the belongsTo property. If you tried the same code with organization -> users instead, it would work.
The fix is rather odd: make the users collection on Organization non-lazy too.
This one will have to make it into the GORM Gotchas series!