I devloppe a batch with juste main method in legacy project with : java 7, hibernate and Spring using mysql database.
In this batch I want to update a several lines in a table that have more than 50 millions lines.
When I start the batch each day I have to update at least 10000 lines.
So, what is the best way to update the line without lock the table in mysql ?
Juste do one query like this :
update table items set is_achive = true where id in (id1,id2,id3....id10000)
Or use a for loop like this :
for(item p : ItemsList){
update table item set is_achive = true where id = p.id
}
This depends how you determine the list of rows that need updating. If you query the database to determine the list, it's probably best just to use a DML statement like:
UPDATE Item i SET i.achive = true WHERE ...
If your concern is locking i.e. the amount of time rows are locked, you can use batching by using a cursor e.g. some id of the data source.
SELECT id FROM ... WHERE id >= :start AND ...
ORDER BY id
OFFSET 100 -- use a batch size that suites your needs
LIMIT 1 -- use a batch size that suites your needs
The limit and for update can be implemented by using a query
Integer end = entityManager.createQuery("SELECT id FROM ... WHERE id >= :start AND ... ORDER BY id")
.setParameter("start", previousEnd)
.setFirstResult(100) // batch size
.setMaxResults(1)
.getResultList().stream().findFirst().orElse(null);
Then do a query like this
UPDATE Item i SET i.achive = true WHERE i.id BETWEEN :start AND :end
or if the end is null i.e. the last batch use
UPDATE Item i SET i.achive = true WHERE i.id >= :start
Use Hibernate Criteria builder:
CriteriaBuilder cb = this.em.getCriteriaBuilder();
// create update
CriteriaUpdate<Order> update = cb.createCriteriaUpdate(Order.class);
// set the root class
Root e = update.from(Order.class);
// set update and where clause
update.set("amount", newAmount);
update.where(cb.greaterThanOrEqualTo(e.get("amount"), oldAmount));
// perform update
this.em.createQuery(update).executeUpdate();
https://thorben-janssen.com/criteria-updatedelete-easy-way-to/
It is best to try to get to the root if it like Chris B suggested, but if it's something you can't do, then you might also consider leveraging Spring JDBC Batch Update Operations as documented here. These have existed for some time, so find whichever documentation is appropriate for the version you're using.
https://docs.spring.io/spring-framework/docs/3.0.0.M4/reference/html/ch12s04.html
I've recently started learning to use myBatis.I am now facing such a scenario, I need to constantly fetch a new list of Objects through WebService, then for this list, I need to insert/update each object into the oracle DB table through myBatis.
The tricky part is, I cannot simply do a batch insert every time, because some of the objects might already exist in DB, for these records, I need to update the fields of them instead of a new insertion.
My current solution might be very stupid, using Java, build the list of Object from webservice, loop through each of them, do a myBatis select, if it is not a null(already exists in the db), then do a myBatis update; otherwise, do a myBatis insert for this new object.
The function is achieved. But my technical lead says it is very low-efficient, since doing a for loop using Java and insert/update one by one will consume a lot of system resource. He advised me to do batch insert using myBatis by passing a list of objects in.
Batch insertion in myBatis is straightforward, however, since I am not purely inserting(for existing records I need to do update), I don't think batch insert is appropriate here. I've googled a while for this, and realized maybe I will need to use "merge" instead of "insert" (for Oracle).
The examples I googled out for merge in myBatis is only for one object, not in a batch. Thus I want to find out whether experts could offer me some examples on how to do a batch-merge in MyBatis( The correct way to write a Mapper)?
The accepted answer is not the recommended way of handling batch operations. It does not show true batch statements since the batch executor mode should be used when opening a session. See this post in which a code contributor recommended that the proper way to batch update (or insert) is to open a session in batch mode and repeatedly call update (or insert) for a single record.
Here's what works for me:
public void updateRecords(final List<GisObject> objectsToUpdate) {
final SqlSession sqlSession = MyBatisUtils.getSqlSessionFactory().openSession(ExecutorType.BATCH);
try {
final GisObjectMapper mapper = sqlSession.getMapper(GisObjectMapper.class);
for (final GisObject gisObject : objectsToUpdate) {
mapper.updateRecord(gisObject);
}
sqlSession.commit();
} finally {
sqlSession.close();
}
}
Do not use foreach in your update/insert and ensure that it only updates/inserts a single record. I was running into unsolvable oracle errors by doing it according to the accepted answer (invalid character, statement not ended, etc.). As the linked post indicates, the update (or insert) shown in the accepted answer is actually just a giant sql statement.
In my case also there is same scenario. I used for loop to check whether this record exists in databse or not and then according to that I added this object in to two arraylist for insert or update.
And then used batch for insert and update after for loop for that to list.
here is ex. for update according to different where condition
1] this is for update
<foreach collection="attendingUsrList" item="model" separator=";">
UPDATE parties SET attending_user_count = #{model.attending_count}
WHERE fb_party_id = #{model.eid}
</foreach>
2] this is for insert
<insert id="insertAccountabilityUsers" parameterType="AccountabilityUsersModel" useGeneratedKeys="false">
INSERT INTO accountability_users
(
accountability_user_id, accountability_id, to_username,
record_status, created_by, created_at, updated_by, updated_at
)
VALUES
<foreach collection="usersList" item="model" separator=",">
(
#{model.accountabilityUserId}, #{model.accountabilityId}, #{model.toUsername},
'A', #{model.createdBy}, #{model.createdAt}, #{model.updatedBy}, #{model.updatedAt}
)
</foreach>
</insert>
In dao method declare as
void insertAccountabilityUsers(#Param("usersList") List<AccountabilityUsersModel> usersList);
Update
Here is my batch session code
public static synchronized SqlSession getSqlBatchSession() {
ConnectionBuilderAction connection = new ConnectionBuilderAction();
sf = connection.getConnection();
SqlSession session = sf.openSession(ExecutorType.BATCH);
return session;
}
SqlSession session = ConnectionBuilderAction.getSqlSession();
Actually I already given full example here for this question
In oracle if you want to execute multiple statements at one time you have to enclose your statements in "begin" and "end" block. So try to add attributes to foreach as below. This will definitely work.
<foreach collection="customerList" item="object" open="begin" close=";end;" separator=";">
UPDATE customer SET isActive = #{object.isactive}
WHERE customerId= #{object.customerId}
</foreach>
I get following hibernate error. I am able to identify the function which causes the issue. Unfortunately there are several DB calls in the function. I am unable to find the line which causes the issue since hibernate flush the session at the end of the transaction. The below mentioned hibernate error looks like a general error. It doesn't even mentioned which Bean causes the issue. Anyone familiar with this hibernate error?
org.hibernate.StaleStateException: Batch update returned unexpected row count from update: 0 actual row count: 0 expected: 1
at org.hibernate.jdbc.BatchingBatcher.checkRowCount(BatchingBatcher.java:93)
at org.hibernate.jdbc.BatchingBatcher.checkRowCounts(BatchingBatcher.java:79)
at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:58)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:195)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:235)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:142)
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:297)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:985)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:333)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:584)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransacti
onManager.java:500)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManag
er.java:473)
at org.springframework.transaction.interceptor.TransactionAspectSupport.doCommitTransactionAfterReturning(Transaction
AspectSupport.java:267)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:176)
I got the same exception while deleting a record by Id that does not exists at all. So check that record you are updating/Deleting actually exists in DB
Without code and mappings for your transactions, it'll be next to impossible to investigate the problem.
However, to get a better handle as to what causes the problem, try the following:
In your hibernate configuration, set hibernate.show_sql to true. This should show you the SQL that is executed and causes the problem.
Set the log levels for Spring and Hibernate to DEBUG, again this will give you a better idea as to which line causes the problem.
Create a unit test which replicates the problem without configuring a transaction manager in Spring. This should give you a better idea of the offending line of code.
Solution:
In the Hibernate mapping file for the id property, if you use any generator class, for that property you should not set the value explicitly by using a setter method.
If you set the value of the Id property explicitly, it will lead the error above. Check this to avoid this error.
or
It's error show when you mention in the mapping file the field generator="native" or "incremental" and in your DATABASE the table mapped is not auto_incremented
Solution: Go to your DATABASE and update your table to set auto_increment
In my case, I came to this exception in two similar cases:
In a method annotated with #Transactional I had a call to another service (with long times of response). The method updates some properties of the entity (after the method, the entity still exists in the database). If the user requests two times the method (as he thinks it doesn't work the first time) when exiting from the transactional method the second time, Hibernate tries to update an entity which already changed its state from the beginning of the transaction. As Hibernate search for an entity in a state, and found the same entity but already changed by the first request, it throws an exception as it can't update the entity. It's like a conflict in GIT.
I had automatic requests (for monitoring the platform) which update an entity (and the manual rollback a few seconds later). But this platform is already used by a test team. When a tester performs a test in the same entity as the automatic requests, (within the same hundredth of a millisecond), I get the exception. As in the previous case, when exiting from the second transaction, the entity previously fetched already changed.
Conclusion: in my case, it wasn't a problem which can be found in the code. This exception is thrown when Hibernate founds that the entity first fetched from the database changed during the current transaction, so it can't flush it to the database as Hibernate doesn't know which is the correct version of the entity: the one the current transaction fetch at the beginning; or the one already stored in the database.
Solution: to solve the problem, you will have to play with the Hibernate LockMode to find the one which best fit your requirements.
This happened to me once by accident when I was assigning specific IDs to some objects (testing) and then I was trying to save them in the database. The problem was that in the database there was an specific policy for setting up the IDs of the objects. Just do not assign an ID if you have a policy at Hibernate level.
I just encountered this problem and found out I was deleting a record and trying to update it afterwards in a Hibernate transaction.
Hibernate 5.4.1 and HHH-12878 issue
Prior to Hibernate 5.4.1, the optimistic locking failure exceptions (e.g., StaleStateException or OptimisticLockException) didn't include the failing statement.
The HHH-12878 issue was created to improve Hibernate so that when throwing an optimistic locking exception, the JDBC PreparedStatement implementation is logged as well:
if ( expectedRowCount > rowCount ) {
throw new StaleStateException(
"Batch update returned unexpected row count from update ["
+ batchPosition + "]; actual row count: " + rowCount
+ "; expected: " + expectedRowCount + "; statement executed: "
+ statement
);
}
Testing Time
I created the BatchingOptimisticLockingTest in my High-Performance Java Persistence GitHub repository to demonstrate how the new behavior works.
First, we will define a Post entity that defines a #Version property, therefore enabling the implicit optimistic locking mechanism:
#Entity(name = "Post")
#Table(name = "post")
public class Post {
#Id
#GeneratedValue(strategy = GenerationType.SEQUENCE)
private Long id;
private String title;
#Version
private short version;
public Long getId() {
return id;
}
public Post setId(Long id) {
this.id = id;
return this;
}
public String getTitle() {
return title;
}
public Post setTitle(String title) {
this.title = title;
return this;
}
public short getVersion() {
return version;
}
}
We will enable the JDBC batching using the following 3 configuration properties:
properties.put("hibernate.jdbc.batch_size", "5");
properties.put("hibernate.order_inserts", "true");
properties.put("hibernate.order_updates", "true");
We are going to create 3 Post entities:
doInJPA(entityManager -> {
for (int i = 1; i <= 3; i++) {
entityManager.persist(
new Post()
.setTitle(String.format("Post no. %d", i))
);
}
});
And Hibernate will execute a JDBC batch insert:
SELECT nextval ('hibernate_sequence')
SELECT nextval ('hibernate_sequence')
SELECT nextval ('hibernate_sequence')
Query: [
INSERT INTO post (title, version, id)
VALUES (?, ?, ?)
],
Params:[
(Post no. 1, 0, 1),
(Post no. 2, 0, 2),
(Post no. 3, 0, 3)
]
So, we know that JDBC batching works just fine.
Now, let's replicate the optimistic locking issue:
doInJPA(entityManager -> {
List<Post> posts = entityManager.createQuery("""
select p
from Post p
""", Post.class)
.getResultList();
posts.forEach(
post -> post.setTitle(
post.getTitle() + " - 2nd edition"
)
);
executeSync(
() -> doInJPA(_entityManager -> {
Post post = _entityManager.createQuery("""
select p
from Post p
order by p.id
""", Post.class)
.setMaxResults(1)
.getSingleResult();
post.setTitle(post.getTitle() + " - corrected");
})
);
});
The first transaction selects all Post entities and modifies the title properties.
However, before the first EntityManager is flushed, we are going to execute a second transition using the executeSync method.
The second transaction modifies the first Post, so its version is going to be incremented:
Query:[
UPDATE
post
SET
title = ?,
version = ?
WHERE
id = ? AND
version = ?
],
Params:[
('Post no. 1 - corrected', 1, 1, 0)
]
Now, when the first transaction tries to flush the EntityManager, we will get the OptimisticLockException:
Query:[
UPDATE
post
SET
title = ?,
version = ?
WHERE
id = ? AND
version = ?
],
Params:[
('Post no. 1 - 2nd edition', 1, 1, 0),
('Post no. 2 - 2nd edition', 1, 2, 0),
('Post no. 3 - 2nd edition', 1, 3, 0)
]
o.h.e.j.b.i.AbstractBatchImpl - HHH000010: On release of batch it still contained JDBC statements
o.h.e.j.b.i.BatchingBatch - HHH000315: Exception executing batch [
org.hibernate.StaleStateException:
Batch update returned unexpected row count from update [0];
actual row count: 0;
expected: 1;
statement executed:
PgPreparedStatement [
update post set title='Post no. 3 - 2nd edition', version=1 where id=3 and version=0
]
],
SQL: update post set title=?, version=? where id=? and version=?
So, you need to upgrade to Hibernate 5.4.1 or newer to benefit from this improvement.
This can happen when trigger(s) execute additional DML (data modification) queries which affect the row counts. My solution was to add the following at the top of my trigger:
SET NOCOUNT ON;
I was facing same issue.
The code was working in the testing environment. But it was not working in staging environment.
org.hibernate.jdbc.BatchedTooManyRowsAffectedException: Batch update returned unexpected row count from update [0]; actual row count: 3; expected: 1
The problem was the table had single entry for each primary key in testing DB table. But in staging DB there was multiple entry for same primary key. ( Problem is in staging DB the table didn't had any primary key constraints also there was multiple entry.)
So every time on update operation it gets failed. It tries to update single record and expect to get update count as 1. But since there was 3 records in the table for the same primary key, The result update count finds 3. Since expected update count and actual result update count didn't match, It throws exception and rolls back.
After the I removed all the records which have duplicate primary key and added primary key constraints. It is working fine.
Hibernate - Batch update returned unexpected row count from update: 0 actual row count: 0 expected: 1
actual row count: 0 // means no record found to update
update: 0 // means no record found so nothing update
expected: 1 // means expected at least 1 record with key in db table.
Here the problem is that the query trying to update a record for some key, But hibernate didn't find any record with the key.
It also can happen when you try to UPDATE a PRIMARY KEY.
My two cents.
Problem: With Spring Boot 2.7.1 the h2 database version has changed to v2.1.214 which may result into a thrown OptimisticLockException when using generated UUIDs for Id columns, see https://hibernate.atlassian.net/browse/HHH-15373.
Solution: Add columnDefinition="UUID" to the #Column annotation
E.g., with a primary key definition for an entity like this:
#Id
#GeneratedValue(generator = "UUID")
#GenericGenerator(name = "UUID", strategy = "org.hibernate.id.UUIDGenerator")
#Column(name = COLUMN_UUID, updatable = false, nullable = false)
UUID uUID;
Change the column annotation to:
#Column(name = COLUMN_UUID, updatable = false, nullable = false, columnDefinition="UUID")
As Julius says this happens when an update Occurs on an Object that has its children being deleted. (Probably because there was a need for an update for the whole Father Object and sometimes we prefer to delete the children and re -insert them on the Father (new , old doesnt matter )along with any other updates the father could have on any of its other plain fields)
So ...in order for this to work delete the children (within a Transaction) by calling childrenList.clear() (Dont loop through the children and delete each one with some childDAO.delete(childrenList.get(i).delete())) and setting
#OneToMany(cascade = CascadeType.XXX ,orphanRemoval=true) on the Side of the Father Object. Then update the father (fatherDAO.update(father)). (Repeat for every father object) The result is that children have their link to their father stripped off and then they are being removed as orphans by the framework.
I encountered this problem where we had one-many relationship.
In the hibernate hbm mapping file for master, for object with set type arrangement, added cascade="save-update" and it worked fine.
Without this, by default hibernate tries to update for a non-existent record and by doing so it inserts instead.
Another way to get this error is if you have a null item in a collection.
It happens when you try to delete an object and then you try to update the same object. Use this after delete:
session.clear();
i got the same problem and i verified this may occur because of Auto increment primary key. To solve this problem do not inset auto increment value with data set. Insert data without the primary key.
This happened to me too, because I had my id as Long, and I was receiving from the view the value 0, and when I tried to save in the database I got this error, then I fixed it by set the id to null.
This problem mainly occurs when we are trying to save or update the object which are already fetched into memory by a running session.
If you've fetched object from the session and you're trying to update in the database, then this exception may be thrown.
I used session.evict(); to remove the cache stored in hibernate first or if you don't wanna take risk of loosing data, better you make another object for storing the data temp.
try
{
if(!session.isOpen())
{
session=EmployeyDao.getSessionFactory().openSession();
}
tx=session.beginTransaction();
session.evict(e);
session.saveOrUpdate(e);
tx.commit();;
EmployeyDao.shutDown(session);
}
catch(HibernateException exc)
{
exc.printStackTrace();
tx.rollback();
}
I ran into this issue when I was manually beginning and committing transactions inside of method annotated as #Transactional. I fixed the problem by detecting if an active transaction already existed.
//Detect underlying transaction
if (session.getTransaction() != null && session.getTransaction().isActive()) {
myTransaction = session.getTransaction();
preExistingTransaction = true;
} else {
myTransaction = session.beginTransaction();
}
Then I allowed Spring to handle committing the transaction.
private void finishTransaction() {
if (!preExistingTransaction) {
try {
tx.commit();
} catch (HibernateException he) {
if (tx != null) {
tx.rollback();
}
log.error(he);
} finally {
if (newSessionOpened) {
SessionFactoryUtils.closeSession(session);
newSessionOpened = false;
maxResults = 0;
}
}
}
}
This happens when you declared the JSF Managed Bean as
#RequestScoped;
when you should declare as
#SessionScoped;
Regards;
I got this error when I tried to update an object with an id that did not exist in the database. The reason for my mistake was that I had manually assigned a property with the name 'id' to the client side JSON-representation of the object and then when deserializing the object on the server side this 'id' property would overwrite the instance variable (also called 'id') that Hibernate was supposed to generate. So be careful of naming collisions if you are using Hibernate to generate identifiers.
I also came across the same challenge. In my case I was updating an object which was not even existing, using hibernateTemplate.
Actually in my application I was getting a DB object to update. And while updating its values, I also updated its ID by mistake, and went ahead to update it and came across the said error.
I am using hibernateTemplate for CRUD operations.
After reading all answers did´t find anyone to talk about inverse atribute of hibernate.
In my my opinion you should also verify in your relationships mapping whether inverse key word is appropiately setted. Inverse keyword is created to defines which side is the owner to maintain the relationship. The procedure for updating and inserting varies cccording to this attribute.
Let's suppose we have two tables:
principal_table, middle_table
with a relationship of one to many. The hiberntate mapping classes are Principal and Middle respectively.
So the Principal class has a SET of Middle objects. The xml mapping file should be like following:
<hibernate-mapping>
<class name="path.to.class.Principal" table="principal_table" ...>
...
<set name="middleObjects" table="middle_table" inverse="true" fetch="select">
<key>
<column name="PRINCIPAL_ID" not-null="true" />
</key>
<one-to-many class="path.to.class.Middel" />
</set>
...
As inverse is set to ”true”, it means “Middle” class is the relationship owner, so Principal class will NOT UPDATE the relationship.
So the procedure for updating could be implemented like this:
session.beginTransaction();
Principal principal = new Principal();
principal.setSomething("1");
principal.setSomethingElse("2");
Middle middleObject = new Middle();
middleObject.setSomething("1");
middleObject.setPrincipal(principal);
principal.getMiddleObjects().add(middleObject);
session.saveOrUpdate(principal);
session.saveOrUpdate(middleObject); // NOTICE: you will need to save it manually
session.getTransaction().commit();
This worked for me, bu you can suggest some editions in order to improve the solution. That way we all will be learning.
In our case we finally found out the root cause of StaleStateException.
In fact we were deleting the row twice in a single hibernate session. Earlier we were using ojdbc6 lib, and this was ok in this version.
But when we upgraded to odjc7 or ojdbc8, deleting records twice was throwing exception. There was bug in our code where we were deleting twice, but that was not evident in ojdbc6.
We were able to reproduce with this piece of code:
Detail detail = getDetail(Long.valueOf(1396451));
session.delete(detail);
session.flush();
session.delete(detail);
session.flush();
On first flush hibernate goes and makes changes in database. During 2nd flush hibernate compares session's object with actual table's record, but could not find one, hence the exception.
I solved it. I found that there was no primary key for my Id column in table.
Once I created it solved for me. Also there was duplicate id found in table before which I deleted and solved it.
This thread is a bit old, however I thought I should drop my fix here in case it may help someone with same root cause.
I was migrating a Java Spring hibernate app. from Oracle to Postgre, along the migration process, I converted a trigger from Oracle to Postgre, the trigger was "on Before Insert" of a table and was setting a one of the columns value (of course the desired column was marked update=false insert=false in hibernate mapping to allow the trigger to set its value), and when inserting data from the application I got this error Hibernate - Batch update returned unexpected row count from update: 0 actual row count: 0 expected: 1
My mistake was that I was setting "Return NULL" at the end of the trigger function, so when the trigger set the column value and the control is back to hibernate for saving, the record was lost as I was returning null.
My fix was to change "Return NULL" to "RETURN NEW" in trigger, this will keep the record available after being altered by the trigger, simply this was what it means by "unexcepted row count for update: 0 expected 1"
This happened if you change something in data set using native sql query but persisted object for same data set is present in session cache.
Use session.evict(yourObject);
Hibernate caches objects from the session. If object is accessed and modified by more than 1 user then org.hibernate.StaleStateException may be be thrown. It may be solved with merge/refresh entity method before saving or using lock. More info: http://java-fp.blogspot.lt/2011/09/orghibernatestalestateexception-batch.html
One of the case
SessionFactory sf=new Configuration().configure().buildSessionFactory();
Session session=sf.openSession();
UserDetails user=new UserDetails();
session.beginTransaction();
user.setUserName("update user agian");
user.setUserId(12);
session.saveOrUpdate(user);
session.getTransaction().commit();
System.out.println("user::"+user.getUserName());
sf.close();
I was facing this exception, and hibernate was working well. I tried to insert manually one record using pgAdmin, here the issue became clear. SQL insert query returns 0 insert. and there is a trigger function that cause this issue because it returns null. so I have only to set it to return new.
and finally I solved the problem.
hope that helps any body.