Develop Reference

What is the use of Hibernate batch processing

I am new to hibernate i have doubt in hibernate batch processing, i read some tutorial for hibernate batch processing they said
Session session = SessionFactory.openSession();
Transaction tx = session.beginTransaction();
for ( int i=0; i<100000; i++ )
{
Employee employee = new Employee(.....);
session.save(employee);
}
tx.commit();
session.close();
Hibernate will cache all the persisted objects in the session-level cache and ultimately your application would fall over with an OutOfMemoryException somewhere around the 50,000th row. You can resolve this problem if you are using batch processing with Hibernate like,
Session session = SessionFactory.openSession();
Transaction tx = session.beginTransaction();
for ( int i=0; i<100000; i++ )
{
Employee employee = new Employee(.....);
session.save(employee);
if( i % 50 == 0 )
{ // Same as the JDBC batch size
//flush a batch of inserts and release memory:
session.flush();
session.clear();
}
}
tx.commit();
session.close();
My doubt is instead of initializing the session outside, why can't we initialize it in to the for loop like,
Session session = null;
Transaction tx = session.beginTransaction();
for ( int i=0; i<100000; i++ )
{
session =SessionFactory.openSession()
Employee employee = new Employee(.....);
session.save(employee);
}
tx.commit();
session.close();
Is it correct way or not any one suggest me the correct way?

No. Don't initialize the session in the for loop; every time you start a new session you're starting a new batch (so you have a batch size of one your way, that is it is non-batching). Also, it would be much slower your way. That is why the first example has
if( i % 50 == 0 ) {
//flush a batch of inserts and release memory:
session.flush();
session.clear();
}
that is what "flush a batch of inserts and release memory" was for.

Batch Processing in Hibernate means to divide a task of huge numbers to some smaller tasks.
When you fire session.save(obj), hibernate will actually cache that object into its memory (still the object is not written into database), and would save it to database when you commit your transaction i.e when you call transactrion.commit().
Lets say you have millions of records to insert, so firing session.save(obj) would consume a lot of memory and eventually would result into OutOfMemoryException.
Solution :
Creating a simple batch of smaller size and saving them to database.
if( i % 50 == 0 ) {
//flush a batch of inserts and release memory:
session.flush();
session.clear();
}
Note :
In code above session.flush() would flush i.e actually save the objects into database and session.clear() would clear any memory occupied by those objects for a batch of size 50.

Batch processing allows you to optimize writing data.
However, the usual advice of flushing and clearing the Hibernate Session is incomplete.
You need to commit the transaction at the end of the batch to avoid long-running transactions which can hurt performance and, if the last item fails, undoing all changes is going to put a lot of pressure on the DB.
Therefore, this is how you should do batch processing:
int entityCount = 50;
int batchSize = 25;
EntityManager entityManager = entityManagerFactory().createEntityManager();
EntityTransaction entityTransaction = entityManager.getTransaction();
try {
entityTransaction.begin();
for (int i = 0; i < entityCount; i++) {
if (i > 0 && i % batchSize == 0) {
entityTransaction.commit();
entityTransaction.begin();
entityManager.clear();
}
Post post = new Post(
String.format("Post %d", i + 1)
);
entityManager.persist(post);
}
entityTransaction.commit();
} catch (RuntimeException e) {
if (entityTransaction.isActive()) {
entityTransaction.rollback();
}
throw e;
} finally {
entityManager.close();
}

Hiberate delete rows from table with limit

Hi want to delete the millions of rows from the table in batch to avoid locking. I am trying below code but its deleting all the rows.
Session session;
try {
session = dao.getHibernateTemplate().getSessionFactory().getCurrentSession();
} catch (HibernateException e) {
session = dao.getHibernateTemplate().getSessionFactory().openSession();
}
String sql = "delete from "+clazz.getSimpleName();
session.createQuery(sql).setFetchSize(limit).executeUpdate();
dao.getHibernateTemplate().flush();
Is there any better way of doing it

I am considering "clazz.getSimpleName();" is returning a table name.
If this is the case than your query is - "delete from 'tablename'" here you are not specifying any condition which restrict the delete statement, that's why it is deleting all the rows from the table.
As you are using setFetchSize - setFetchSize(int value) is a 'hint' to the driver, telling it how many rows it should fetch.
I think this method is not require in case of delete query.

InnoDB database locking blocks threads

I have a really weird issue with a project I'm working with. I would appreciate if someone could point me to a right direction here.
// Setup
There are multiple web servers and a loadbalancer is in front of them. Servers are handling requests that might come in multiple parts and parts can be handled by different servers. These multi-part requests should be combined to a one single transaction that is going forward once all the parts are received.
The server that does the final processing doesn't matter, but only one server can do it. Other servers that receive the previous parts should just mark the part received, store the data and give a immediate response back.
For now I'm using database table to handle the synchronization between nodes.
The basic idea is that when a server gets a part it tries to acquire the lock with a transaction id coming with the rquest. This is done by trying to insert a row to a Lock table with the txid as a primary key. If insert is successful, that server gets the lock and processes the part it received, by storing it to database checks if other parts have been received and returns a response immediately if not.
// The Problem
The problem I have is that the threads seem to randomly lock at the database and thus freezing the whole processing. I have debugged it to the point that in a situation where multiple requests come to processing at the same time they just get stuck at trying to acquire the lock and ultimately timeout after 30 seconds. Few of the first requests might get processed or not it seems to be random but even something like 7 concurrent requests block the database.
To me there should not be any way how this could get stuck and I'm fresh out of ideas.
// Information
I am using MySQL with an InnoDB engine. Servers are running Java code and Hibernate is used as a ORM layer to access the DB.
The Lock table:
CREATE TABLE `lock` (
`id` varchar(255) NOT NULL,
`expiryDate` datetime DEFAULT NULL,
`issueDate` datetime DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
The id is the transaction id used to combine the parts.
I have an basic interface that manages the lock accessing.
public interface LockProviderDao {
public boolean lock(String id);
public boolean unlock(String id);
}
And a implementation of that class that uses Hibernate to access database.
#Override
public boolean lock(String id) {
Session session = this.sessionFactory.openSession();
Lock lock = new Lock(id);
Transaction tx = null;
boolean locked = false;
try {
// Try to lock
tx = session.beginTransaction();
session.save(lock);
tx.commit();
locked = true;
} catch(Exception e) {
if(tx != null) {
tx.rollback();
}
} finally {
session.close();
}
return locked;
}
#Override
public boolean unlock(String id) {
Session session = this.sessionFactory.openSession();
boolean status = true;
Transaction tx = null;
try {
Lock lock = (Lock) session.load(Lock.class, id);
tx = session.beginTransaction();
session.delete(lock);
tx.commit();
} catch(Exception e) {
if(tx != null) {
tx.rollback();
}
status = false;
} finally {
session.close();
}
return status;
}
Seems simple enough. Here is the code that does the processing. This thread has a Hibernate session opened already so the Session opened inside the lock and unlock methods is a nested Session, if that makes any difference.
int counter = 0;
boolean lockAcquired = false;
do {
// Try to acquire the lock
lockAcquired = this.lockProviderDao.lock(txId);
if (!lockAcquired) {
// Didn't get it try a bit later
try {
Thread.sleep(defaultSleepPeriod);
} catch (Exception e) {
}
if (counter >= defaultSleepCycles) {
return;
}
counter++;
}
} while (!lockAcquired);
// DO THE PROCESSING HERE ONCE LOCK ACQUIRED
// Release the lock
this.lockProviderDao.unlock(txId);

I would lock after inserting the data. This means, that you would have to change your algorithm to something like this:
Begin transaction
Insert the fragment to database
Commit transaction
Begin transaction
Count number of framgents inserted / exit, if not equal to expected fragment count
Insert a row, that indicates that fragments will be processed (e.g. your lock row). If this fails, fragments have been processed or are being processed (= rollback)
Commit transaction
Begin transaction
Read fragments (and verify that they still exist)
Process fragments
Delete lock and fragments (verify they still exist)
Commit transaction
If you need to increase reliability, you have three options:
1. Use JMS with JTA to control program flow
2. Have your client poll the server for status and start processing, if all parts have been received, but processing has not started yet or has been stalled
3. Create a scheduler that starts processing, if same conditions apply

JDO / Duplicate entry exception

I get an MySQLIntegrityConstraintViolationException when saving an object to my database. I know what this error means, but I cannot work around it.
Error: Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '12345' for key 'PRIMARY'
Basically, I want to save course objects to a database. Each course object may have several studypath objects, which can in turn be part of several course objects.
PersistenceManager pm = pmf.getPersistenceManager();
Transaction tx = pm.currentTransaction();
try {
tx.begin();
Query query = pm.newQuery(Studypath.class,"studypathID == paramStudypathID");
query.declareParameters("Integer paramStudypathID");
query.setUnique(true);
Studypath dbStudypath = (Studypath)query.execute(12345);
Studypath detachedStudypath = null;
if (dbStudypath != null) {
detachedStudypath = (Studypath)pm.detachCopy(dbStudypath);
} else {
Studypath newStudypath = new Studypath();
// ...
pm.makePersistent(newStudypath);
detachedStudypath = (Studypath)pm.detachCopy(newStudypath);
}
tx.commit();
// now I want to add this detached studypath to my newly created course
Course c = new Course();
c.addStudypath(detachedStudypath);
tx.begin();
pm.makePersistent(c); // <== error
tx.commit();
}
catch (Exception e)
{
//... handle exceptions
}
finally
{
if (tx.isActive())
{
// Error occurred so rollback the transaction
tx.rollback();
}
pm.close();
}
Course.java
#PersistenceCabable
public class Course {
// ...
#Persistent
private Set<Studypath> studypaths;
}
Studypath.java
#PersistenceCabable
public class Studypath {
// ...
#Persistent
#PrimaryKey
private Integer studypathID;
}
Is there any obvious mistake I'm missing? Thanks in advance!
Update (log):
DEBUG [DataNucleus.Datastore.Native] - SELECT 'Courses.Studypath' AS NUCLEUS_TYPE, ... FROM `STUDYPATH` `A0` WHERE `A0`.`STUDYPATHID` = <12345> // this one already exists
DEBUG [DataNucleus.Datastore.Retrieve] - Execution Time = 0 ms
DEBUG [DataNucleus.Datastore.Retrieve] - Retrieving PreparedStatement for connection "jdbc:mysql://127.0.0.1/database, UserName=user, MySQL-AB JDBC Driver"
DEBUG [DataNucleus.Datastore.Native] - SELECT 'Courses.Course' AS NUCLEUS_TYPE, ... FROM `COURSE` `A0` WHERE `A0`.`COURSEID` = <1111> // there is no such course, thus it gets created
DEBUG [DataNucleus.Datastore.Retrieve] - Execution Time = 1 ms
DEBUG [DataNucleus.Datastore.Retrieve] - Retrieving PreparedStatement for connection "jdbc:mysql://127.0.0.1/database, UserName=user, MySQL-AB JDBC Driver"
DEBUG [DataNucleus.Datastore.Native] - INSERT INTO `COURSE` (...,`COURSEID`) VALUES (...,<1111>)
DEBUG [DataNucleus.Datastore.Persist] - Execution Time = 1 ms (number of rows = 1)
DEBUG [DataNucleus.Datastore.Retrieve] - Closing PreparedStatement org.datanucleus.store.rdbms.ParamLoggingPreparedStatement#3baac1b5
DEBUG [DataNucleus.Datastore.Persist] - The requested statement "INSERT INTO `STUDYPATH` (...) VALUES (...)" has been made batchable
DEBUG [DataNucleus.Datastore.Persist] - Batch has been added to statement "INSERT INTO `STUDYPATH` (...) VALUES (...)" for processing (batch size = 1)
DEBUG [DataNucleus.Datastore.Persist] - Adding statement "INSERT INTO `STUDYPATH` (...) VALUES (...)" to the current batch (new batch size = 2)
DEBUG [DataNucleus.Datastore.Persist] - Batch has been added to statement "INSERT INTO `STUDYPATH` (...) VALUES (...)" for processing (batch size = 2)
DEBUG [DataNucleus.Datastore.Native] - BATCH [INSERT INTO `STUDYPATH` (...,`STUDYPATHID`) VALUES (...,<12345>); INSERT INTO `STUDYPATH` (...,`STUDYPATHID`) VALUES (<54321>)]
ERROR [DataNucleus.Datastore] - Exception thrown

I'm not sure it's kosher to associate a detached JDO to a transient one. There's no easy way for the ORM to know the relation is an existing JDO.
If it's really in the same code path, I'd associate the persistent instance:
c.addStudypath(dbStudypath);
Otherwise I would makePersistent(detachedStudypath) before associating it (assuming your class is #Detachable)

You can easily check state of objects by calling JDOHelper.getObjectState(obj). I strongly suggest to you that your object is in TRANSIENT state not DETACHED state, and likely because you haven't declared your class as detachable.

Hibernate is not deleting my objects. Why?

I have just set up a test that checks that I am able to insert entries into my database using Hibernate. The thing that drives me crazy is that Hibernate does not actually delete the entries, although it reports that they are gone!
The test below runs successfully, but when I check my DB afterwards the entries that were inserted are still there! I even try to check it using assert (yes I have -ea as vm parameter). Does anyone have a clue why the entries are not deleted?
public class HibernateExportStatisticDaoIntegrationTest {
HibernateExportStatisticDao dao;
Transaction transaction;
#Before
public void setUp(){
assert numberOfStatisticRowsInDB() == 0;
dao = new HibernateExportStatisticDao(HibernateUtil.getSessionFactory());
}
#After
public void deleteAllEntries(){
assert numberOfStatisticRowsInDB() != 0;
Session session = HibernateUtil.getSessionFactory().getCurrentSession();
for(PersistableStatisticItem item:allStatisticItemsInDB()) {
session.delete(item);
}
session.flush();
assert numberOfStatisticRowsInDB() == 0;
}
#Test public void exportAllSavesEntriesToDatabase(){
int expectedNumberOfStatistics = 20;
dao.exportAll(StatisticItemFactory.createTestStatistics(expectedNumberOfStatistics));
assertEquals(expectedNumberOfStatistics, numberOfStatisticRowsInDB());
}
private int numberOfStatisticRowsInDB() {
return allStatisticItemsInDB().size();
}
#SuppressWarnings("unchecked")
private List<PersistableStatisticItem> allStatisticItemsInDB(){
Session session = HibernateUtil.getSessionFactory().getCurrentSession();
transaction = session.beginTransaction();
Query q = session.createQuery("FROM PersistableStatisticItem item");
return q.list();
}
}
The console is filled with
Hibernate: delete from UPTIME_STATISTICS where logDate=? and serviceId=?
but nothing has been deleted when I check it.

I guess it's related to inconsistent use of transactions (note that beginTransaction() in allStatisticItemsInDB() is called several times without corresponding commits).
Try to manage transactions in proper way, for example, like this:
Session session = HibernateUtil.getSessionFactory().getCurrentSession();
Transaction tx = session.beginTransaction();
for(PersistableStatisticItem item:
session.createQuery("FROM PersistableStatisticItem item").list()) {
session.delete(item);
}
session.flush();
assert session.createQuery("FROM PersistableStatisticItem item").list().size() == 0;
tx.commit();
See also:
13.2. Database transaction demarcation

I have the same problem. Although I was not using transaction at all. I was using namedQuery like this :
Query query = session.getNamedQuery(EmployeeNQ.DELETE_EMPLOYEES);
int rows = query.executeUpdate();
session.close();
It was returning 2 rows but the database still had all the records. Then I wrap up the above code with this :
Transaction transaction = session.beginTransaction();
Query query = session.getNamedQuery(EmployeeNQ.DELETE_EMPLOYEES);
int rows = query.executeUpdate();
transaction.commit();
session.close();
Then it started working fine. I was using SQL server. But I think if we use h2, above code (without transaction) will also work fine.
One more observation : To insert and get records usage of transaction is not mandatory but for deletion of records we will have to use transaction. (only tested in SQL server)

Can you post your DB schema and HBM or Fluent maps? One thing that got me a while back was I had a ReadOnly() in my Fluent map. It never threw an error and I too saw the "delete from blah where blahblah=..." in the logs.