My Spring component gets a request from a client, asks a web-service about some data and saves received objects to a database.
I identify all objects and save only new ones.
The issue occurs when the client makes two or more same requests in the same time (or due to even different user requests I receive same objects from web-service).
To describe the issue with persistence here some details. For each client request my component starts execution in a separate thread, I get a new entityManager, begin a transaction, receive a data from web-service, then I identify objects and persist new ones using given entityManager in a current transaction.
If in separate transactions I receive the same objects from web-service and if they are new ones that are not yet in database I am not able to identify them in not-commited transactions and so they are persisted in all transactions. Then all duplicate objects will be commited and saved to database.
What could be good solutions in this case? Is there any way to identify new objects properly even in different transactions? Or what approaches can be applied?
May be Spring provides some approaches to manage transactions or entityManagers so that it can help with this issue...
Note. Of course I can use database instruments to avoid saving duplicate objects but in this case it is not a very good solution.
Check if objects are present in a database before saving.
Use #UniqueConstraint or #Column(unique = true) to prevent duplicate rows, handle exceptions appropriately.
Use #Version to manage concurrent modification for existing entities. More about optimistic and pesimistic locking: Chapter 5. Locking. Related discussions: Hibernate Automatic Versioning and When to use #Version and #Audited in Hibernate?
You may use thread locks / synchronization mechanisms to ensure that requests for the same user will happen in order. However, this won't work if your service in running on more than 1 node.
So the solution in my case is the following:
Make transactions pretty small and commit every object separately.
Make unique constraints in database to prevent duplicating of
objects. This point will not help us a lot but needed for point 3.
Every commit() method we insert in try-catch block. If we try to
commit duplicate object in parallel transactions then we will receive an exception and in catch block we can check the database, select the object that is already there and work with it futher.
The example:
boolean reidentifyNeed = false;
try {
DofinService.getEntityManagerThreadLocal().getTransaction().begin();
DofinService.getEntityManagerThreadLocal().persist(entity);
try {
DofinService.getEntityManagerThreadLocal().getTransaction().commit();
//if commit is successfull
entityIdInDB = (long) entity.getId();
DofinService.getEntityManagerThreadLocal().clear();
} catch (Exception ex) {
logger.error("Error committing " + entity.getClass().getSimpleName() + " in DB. Possibly duplicate object. Will try to re-identify object. Error: " + ex.toString());
reidentifyNeed = true;
}
if(reidentifyNeed){
//need clear entityManager, because if duplicated object was persisted then during *select* an object flush() method will be executed and it will thrown ConstrainViolationException
DofinService.getEntityManagerThreadLocal().clear();
CheckSimilarObject checkSimilarObject = new CheckSimilarObject();
long objectId = checkSimilarObject.checkObject(dofinObject);
logger.warn("Re-identifying was done. EntityId = " + objectId);
entityIdInDB = objectId;
}
} catch (Exception ex) {
logger.error("Error persisting and commiting object: " + ex.toString());
}
I have a problem with syncronization of a Database field.
I have a Singleton class for instance EntityManager, I use code like this :
Persistence.createEntityManagerFactory("Myproject_EJB")
I create EntityManager using this singleton from both a WebApplication and from WebService layer and use it for access to persistence.
User can modify this Database field using WebApplication, but data is stored in Database only when request is complete (request can take several seconds). Meanwhile if someone call WebService and ask for same field I have an inconsistent state.
In persistence.xml I have use_second_level_cache and use_query_cache set true, and transaction-type="JTA".
In webApp I use code like this for update data :
EntityManager em = EntityMan.getEMF().createEntityManager();
try {
em.find(Tel.class, tel.getTelId());
em.merge(tel);
em.flush();
} catch (Exception e) {
logger.debug(e.getMessage());
e.printStackTrace();
}
How can I solve please ??
don't use caching for data which is changed frequently, or keep cache time very small ? you can set table level caches
http://www.tutorialspoint.com/hibernate/hibernate_caching.htm
just giving pointers to look not a perfect answer.
When the program encounter an exception, The RollBackOnly goes to True.
How can I "Set" this RollBack To False Even it is encountering an exception.
#Resource
protected SessionContext context;
Public void creerNewEntity(final Entity myEntity) {
try {
this.em.persist(myEntity);
this.em.flush();
} catch (final EntityExistsException e) {
System.out.println((this.context.getRollbackOnly())); // It s has a true value
throw new MyException("TODO BLABLA", e);
}
}
When the program throw this Exception "MyException", I change the object myEntity by setting for example a new Id then I call again creerNewEntity().
Unfortunately, it doesn't work, I got this exception "javax.persistence.PersistenceException: org.hibernate.HibernateException: proxy handle is no longer valid", I think because the RollBack has a true value, How can I change the rollback to make this works ?
Thanks.
There probably isn't a simple way to do this since the whole point of the EJB design is that you don't care about such things. The first error inside of a transaction makes it invalid -> rollback. That's the rule.
If you want something special, then get yourself a database connection from the session and use plain SQL instead of EJB to modify the data. That way, you can try to INSERT a new instance and handle all exceptions yourself. When the insert succeeds, you can use EJB to load the newly created object to add it to the session.
That said, I'm not sure what you're trying to achieve with the code above. Just ignoring when you can't create a new instance in the database feels like "I don't care about quality of my product." Maybe your attempt to work around an error is just a symptom of a bad design of your application. Take a step back and consider what you're doing and why. Maybe if you told us more about the reasons why you want to ignore all errors (even the really, really deadly ones), we would be able to point out a better solution.
EDIT So you get javax.persistence.EntityExistsException which means you tried to save the same entity twice. That can mean any number of things:
You loaded the bean in a different session and now you try to save it in a second one. Since the new session can't know if the bean exists, it tries to create it again.
Instead of loading the bean from the session like you should, you cheated and create a new instance manually. Of course, the session manager now thinks this is a new bean.
The correct solution depends on what you need to achieve. If you modified myEntity and need to save the changes, use em.merge(). The EM will then check if the object already exists and if it does, it will do an SQL UPDATE instead of an INSERT
If you just want to give some other class a valid entity, then you need to get it from the database. If the database returns null, you need to create a new instance and persist it and then return it.
See also: JPA EntityManager: Why use persist() over merge()?
EntityExistsException is PersistenceException
when jpa throws it, ejb CMT is marked for rollback
http://piotrnowicki.com/2013/03/jpa-and-cmt-why-catching-persistence-exception-is-not-enough/
as aaron suggested, you could use merge()
you can also contain transaction boundary by using RequiresNew
#TransactionAttribute(REQUIRES_NEW)
http://docs.oracle.com/javaee/6/tutorial/doc/bncij.html
My current project is done using JavaFX. I use properties to bind (bidirectionnal) view fields to bean (with BeanPathAdapter of JFXtras).
I choose to use JPA with ObjectDB as model.
This is the first time I use JPA in a standalone project and here I'm facing the problem of managed entities.
Actually, I bind managed entities to view fields and when the value of a view field changes, the entities is updated... and the database also.
I'm trying to find a way to manually persist/merge an entity so I can ask the user if he wants to save or not.
Here's the code i use to get list :
EntityManagerFactory emf = Persistence.createEntityManagerFactory("$objectdb/data/db.odb");
EntityManager em = emf.createEntityManager();
List<XXX> entities = em.createQuery("SELECT x FROM XXX x").getResultList();
So when i do
entity.setName("test");
the entity is updated in the database.
What i'm looking for is that the entity doesn't update automatically.
I tried (just after the getResultList)
em.clear();
or
em.detach(entity);
but it looses the relations instances even with CascadeType.DETACH.
I also tried
em.setFlushMode(FlushModeType.COMMIT);
but it still updates automatically...
I also tried to clone the object. But when i want to merge it, it gives me an exception :
Attempt to reuse an existing primary key value
I thought an alternative solution : use a variable as 'buffer' and fill the managed bean with buffer if the user saves. But BeanPathAdapter looses its sense. It's the same as filling view fields manually and filling bean fields manually after saving.
Could you help me to find a solution ?
Thanks,
Smoky
EDIT:
I answer to my own question :p
After 3 hours of research, I found a solution.
The 'cloning' solution was the 'best' of each I quoted but I don't think it's the best one.
The cause of the exception was the code I used to persist/merge my entity. I was persisting an entity non-managed with an already existing id. I thought I was merging...
I did a generic method not to fail again
public <T extends IEntity> T persist(T object) {
em.getTransaction().begin();
if (object.getId() == null) {
em.persist(object);
em.flush();
em.getTransaction().commit();
em.refresh(object);
}
else {
object = em.merge(object);
em.getTransaction().commit();
}
return object;
}
So the solution : When I have to bind the entity to the view, I use entity.clone() so I can use the entity as non-managed and merge when I want.
But if you have a proper solution, i'm interested :)
Thanks again
In addition to the solution above, standard solutions are:
Use detached objects in the model and then merge them into the EntityManager.
Use managed objects in the model, keeping the EntityManager open (with no detach/merge).
I'm trying to write a method that will return a Hibernate object based on a unique but non-primary key. If the entity already exists in the database I want to return it, but if it doesn't I want to create a new instance and save it before returning.
UPDATE: Let me clarify that the application I'm writing this for is basically a batch processor of input files. The system needs to read a file line by line and insert records into the db. The file format is basically a denormalized view of several tables in our schema so what I have to do is parse out the parent record either insert it into the db so I can get a new synthetic key, or if it already exists select it. Then I can add additional associated records in other tables that have foreign keys back to that record.
The reason this gets tricky is that each file needs to be either totally imported or not imported at all, i.e. all inserts and updates done for a given file should be a part of one transaction. This is easy enough if there's only one process that's doing all the imports, but I'd like to break this up across multiple servers if possible. Because of these constraints I need to be able to stay inside one transaction, but handle the exceptions where a record already exists.
The mapped class for the parent records looks like this:
#Entity
public class Foo {
#Id
#GeneratedValue(strategy = IDENTITY)
private int id;
#Column(unique = true)
private String name;
...
}
My initial attempt at writting this method is as follows:
public Foo findOrCreate(String name) {
Foo foo = new Foo();
foo.setName(name);
try {
session.save(foo)
} catch(ConstraintViolationException e) {
foo = session.createCriteria(Foo.class).add(eq("name", name)).uniqueResult();
}
return foo;
}
The problem is when the name I'm looking for exists, an org.hibernate.AssertionFailure exception is thrown by the call to uniqueResult(). The full stack trace is below:
org.hibernate.AssertionFailure: null id in com.searchdex.linktracer.domain.LinkingPage entry (don't flush the Session after an exception occurs)
at org.hibernate.event.def.DefaultFlushEntityEventListener.checkId(DefaultFlushEntityEventListener.java:82) [hibernate-core-3.6.0.Final.jar:3.6.0.Final]
at org.hibernate.event.def.DefaultFlushEntityEventListener.getValues(DefaultFlushEntityEventListener.java:190) [hibernate-core-3.6.0.Final.jar:3.6.0.Final]
at org.hibernate.event.def.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:147) [hibernate-core-3.6.0.Final.jar:3.6.0.Final]
at org.hibernate.event.def.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:219) [hibernate-core-3.6.0.Final.jar:3.6.0.Final]
at org.hibernate.event.def.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:99) [hibernate-core-3.6.0.Final.jar:3.6.0.Final]
at org.hibernate.event.def.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:58) [hibernate-core-3.6.0.Final.jar:3.6.0.Final]
at org.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:1185) [hibernate-core-3.6.0.Final.jar:3.6.0.Final]
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1709) [hibernate-core-3.6.0.Final.jar:3.6.0.Final]
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:347) [hibernate-core-3.6.0.Final.jar:3.6.0.Final]
at org.hibernate.impl.CriteriaImpl.uniqueResult(CriteriaImpl.java:369) [hibernate-core-3.6.0.Final.jar:3.6.0.Final]
Does anyone know what is causing this exception to be thrown? Does hibernate support a better way of accomplishing this?
Let me also preemptively explain why I'm inserting first and then selecting if and when that fails. This needs to work in a distributed environment so I can't synchronize across the check to see if the record already exists and the insert. The easiest way to do this is to let the database handle this synchronization by checking for the constraint violation on every insert.
I had a similar batch processing requirement, with processes running on multiple JVMs. The approach I took for this was as follows. It is very much like jtahlborn's suggestion. However, as vbence pointed out, if you use a NESTED transaction, when you get the constraint violation exception, your session is invalidated. Instead, I use REQUIRES_NEW, which suspends the current transaction and creates a new, independent transaction. If the new transaction rolls back it will not affect the original transaction.
I am using Spring's TransactionTemplate but I'm sure you could easily translate it if you do not want a dependency on Spring.
public T findOrCreate(final T t) throws InvalidRecordException {
// 1) look for the record
T found = findUnique(t);
if (found != null)
return found;
// 2) if not found, start a new, independent transaction
TransactionTemplate tt = new TransactionTemplate((PlatformTransactionManager)
transactionManager);
tt.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRES_NEW);
try {
found = (T)tt.execute(new TransactionCallback<T>() {
try {
// 3) store the record in this new transaction
return store(t);
} catch (ConstraintViolationException e) {
// another thread or process created this already, possibly
// between 1) and 2)
status.setRollbackOnly();
return null;
}
});
// 4) if we failed to create the record in the second transaction, found will
// still be null; however, this would happy only if another process
// created the record. let's see what they made for us!
if (found == null)
found = findUnique(t);
} catch (...) {
// handle exceptions
}
return found;
}
You need to use UPSERT or MERGE to achieve this goal.
However, Hibernate does not offer support for this construct, so you need to use jOOQ instead.
private PostDetailsRecord upsertPostDetails(
DSLContext sql, Long id, String owner, Timestamp timestamp) {
sql
.insertInto(POST_DETAILS)
.columns(POST_DETAILS.ID, POST_DETAILS.CREATED_BY, POST_DETAILS.CREATED_ON)
.values(id, owner, timestamp)
.onDuplicateKeyIgnore()
.execute();
return sql.selectFrom(POST_DETAILS)
.where(field(POST_DETAILS.ID).eq(id))
.fetchOne();
}
Calling this method on PostgreSQL:
PostDetailsRecord postDetailsRecord = upsertPostDetails(
sql,
1L,
"Alice",
Timestamp.from(LocalDateTime.now().toInstant(ZoneOffset.UTC))
);
Yields the following SQL statements:
INSERT INTO "post_details" ("id", "created_by", "created_on")
VALUES (1, 'Alice', CAST('2016-08-11 12:56:01.831' AS timestamp))
ON CONFLICT DO NOTHING;
SELECT "public"."post_details"."id",
"public"."post_details"."created_by",
"public"."post_details"."created_on",
"public"."post_details"."updated_by",
"public"."post_details"."updated_on"
FROM "public"."post_details"
WHERE "public"."post_details"."id" = 1
On Oracle and SQL Server, jOOQ will use MERGE while on MySQL it will use ON DUPLICATE KEY.
The concurrency mechanism is ensured by the row-level locking mechanism employed when inserting, updating, or deleting a record, which you can view in the following diagram:
Code avilable on GitHub.
Two solution come to mind:
That's what TABLE LOCKS are for
Hibernate does not support table locks, but this is the situation when they come handy. Fortunately you can use native SQL thru Session.createSQLQuery(). For example (on MySQL):
// no access to the table for any other clients
session.createSQLQuery("LOCK TABLES foo WRITE").executeUpdate();
// safe zone
Foo foo = session.createCriteria(Foo.class).add(eq("name", name)).uniqueResult();
if (foo == null) {
foo = new Foo();
foo.setName(name)
session.save(foo);
}
// releasing locks
session.createSQLQuery("UNLOCK TABLES").executeUpdate();
This way when a session (client connection) gets the lock, all the other connections are blocked until the operation ends and the locks are released. Read operations are also blocked for other connections, so needless to say use this only in case of atomic operations.
What about Hibernate's locks?
Hibernate uses row level locking. We can not use it directly, because we can not lock non-existent rows. But we can create a dummy table with a single record, map it to the ORM, then use SELECT ... FOR UPDATE style locks on that object to synchronize our clients. Basically we only need to be sure that no other clients (running the same software, with the same conventions) will do any conflicting operations while we are working.
// begin transaction
Transaction transaction = session.beginTransaction();
// blocks until any other client holds the lock
session.load("dummy", 1, LockOptions.UPGRADE);
// virtual safe zone
Foo foo = session.createCriteria(Foo.class).add(eq("name", name)).uniqueResult();
if (foo == null) {
foo = new Foo();
foo.setName(name)
session.save(foo);
}
// ends transaction (releasing locks)
transaction.commit();
Your database has to know the SELECT ... FOR UPDATE syntax (Hibernate is goig to use it), and of course this only works if all your clients has the same convention (they need to lock the same dummy entity).
The Hibernate documentation on transactions and exceptions states that all HibernateExceptions are unrecoverable and that the current transaction must be rolled back as soon as one is encountered. This explains why the code above does not work. Ultimately you should never catch a HibernateException without exiting the transaction and closing the session.
The only real way to accomplish this it would seem would be to manage the closing of the old session and reopening of a new one within the method itself. Implementing a findOrCreate method which can participate in an existing transaction and is safe within a distributed environment would seem to be impossible using Hibernate based on what I have found.
The solution is in fact really simple. First perform a select using your name value. If a result is found, return that. If not, create a new one. In case the creation fail (with an exception), this is because another client added this very same value between your select and your insert statement. This is then logical that you have an exception. Catch it, rollback your transaction and run the same code again. Because the row already exist, the select statement will find it and you'll return your object.
You can see here explanation of strategies for optimistic and pessimistic locking with hibernate here : http://docs.jboss.org/hibernate/core/3.3/reference/en/html/transactions.html
a couple people have mentioned different parts of the overall strategy. assuming that you generally expect to find an existing object more often than you create a new object:
search for existing object by name. if found, return
start nested (separate) transaction
try to insert new object
commit nested transaction
catch any failure from nested transaction, if anything but constraint violation, re-throw
otherwise search for existing object by name and return it
just to clarify, as pointed out in another answer, the "nested" transaction is actually a separate transaction (many databases don't even support true, nested transactions).
Well, here's one way to do it - but it's not appropriate for all situations.
In Foo, remove the "unique = true" attribute on name. Add a timestamp that gets updated on every insert.
In findOrCreate(), don't bother checking if the entity with the given name already exists - just insert a new one every time.
When looking up Foo instances by name, there may be 0 or more with a given name, so you just select the newest one.
The nice thing about this method is that it doesn't require any locking, so everything should run pretty fast. The downside is that your database will be littered with obsolete records, so you may have to do something somewhere else to deal with them. Also, if other tables refer to Foo by its id, then this will screw up those relations.
Maybe you should change your strategy:
First find the user with the name and only if the user thoes not exist, create it.
I would try the following strategy:
A. Start a main transaction (at time 1)
B. Start a sub-transaction (at time 2)
Now, any object created after time 1 will not be visible in the main transaction. So when you do
C. Create new race-condition object, commit sub-transaction
D. Handle conflict by starting a new sub-transaction (at time 3) and getting the object from a query (the sub-transaction from point B is now out-of-scope).
only return the object primary key and then use EntityManager.getReference(..) to obtain the object you will be using in the main transaction. Alternatively, start the main transaction after D; it is not totally clear to me in how many race conditions you will have within your main transaction, but the above should allow for n times B-C-D in a 'large' transaction.
Note that you might want to do multi-threading (one thread per CPU) and then you can probably reduce this issue considerably by using a shared static cache for these kind of conflicts - and point 2 can be kept 'optimistic', i.e. not doing a .find(..) first.
Edit: For a new transaction, you need an EJB interface method call annotated with transaction type REQUIRES_NEW.
Edit: Double check that the getReference(..) works as I think it does.