I'm working on an Integrator for Hibernate (background on Integrators: https://docs.jboss.org/hibernate/orm/4.3/manual/en-US/html/ch14.html#objectstate-decl-security) that by using listeners is supposed to take my data from how it's stored in the DB and convert it into a different form for processing at runtime. This works great when saving the data using .persist() however there's an odd behavior involving transactions. The following code is from Hibernate's own quickstart tutorial code:
// now lets pull events from the database and list them
entityManager = entityManagerFactory.createEntityManager();
entityManager.getTransaction().begin();
List<Event> result = entityManager.createQuery( "from Event", Event.class ).getResultList();
for ( Event event : result ) {
System.out.println( "Event (" + event.getDate() + ") : " + event.getTitle() );
}
entityManager.getTransaction().commit();
entityManager.close();
Notice the unusual transaction begin/commit wrapping the query to select the data. Running this gives the following output after the query completes:
01:01:59.111 [main] DEBUG org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(175) - committing
01:01:59.112 [main] DEBUG org.hibernate.event.internal.AbstractFlushingEventListener.prepareEntityFlushes(149) - Processing flush-time cascades
01:01:59.112 [main] DEBUG org.hibernate.event.internal.AbstractFlushingEventListener.prepareCollectionFlushes(189) - Dirty checking collections
01:01:59.114 [main] DEBUG org.hibernate.event.internal.AbstractFlushingEventListener.logFlushResults(123) - Flushed: 0 insertions, 2 updates, 0 deletions to 2 objects
01:01:59.114 [main] DEBUG org.hibernate.event.internal.AbstractFlushingEventListener.logFlushResults(130) - Flushed: 0 (re)creations, 0 updates, 0 removals to 0 collections
01:01:59.114 [main] DEBUG org.hibernate.internal.util.EntityPrinter.toString(114) - Listing entities:
01:01:59.114 [main] DEBUG org.hibernate.internal.util.EntityPrinter.toString(121) - org.hibernate.tutorial.em.Event{date=2015-07-28 01:01:57.776, id=1, title=Our very first event!}
01:01:59.114 [main] DEBUG org.hibernate.internal.util.EntityPrinter.toString(121) - org.hibernate.tutorial.em.Event{date=2015-07-28 01:01:58.746, id=2, title=A follow up event}
01:01:59.115 [main] DEBUG org.hibernate.SQL.logStatement(109) - update EVENTS set EVENT_DATE=?, title=? where id=?
Hibernate: update EVENTS set EVENT_DATE=?, title=? where id=?
01:01:59.119 [main] DEBUG org.hibernate.SQL.logStatement(109) - update EVENTS set EVENT_DATE=?, title=? where id=?
Hibernate: update EVENTS set EVENT_DATE=?, title=? where id=?
01:01:59.120 [main] DEBUG org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doCommit(113) - committed JDBC Connection
01:01:59.120 [main] DEBUG org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.close(201) - HHH000420: Closing un-released batch
01:01:59.121 [main] DEBUG org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.releaseConnection(246) - Releasing JDBC connection
01:01:59.121 [main] DEBUG org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.releaseConnection(264) - Released JDBC connection
01:01:59.121 [main] DEBUG org.hibernate.internal.SessionFactoryImpl.close(1339) - HHH000031: Closing
It appears that since the Integrator does a modification on the entity in question it gets marked as "dirty" and upon committing this odd transaction, it bypasses my event listeners and writes the value back in the wrong format! I did some digging in the code and it turns out that org.hibernate.event.internal.AbstractFlushingEventListener.flushEntities(FlushEvent, PersistenceContext) gets called above and tries to get listeners for EventType.FLUSH_ENTITY. Unfortunately a listener added for this EventType is never called in my Integrator. How can I write my Integrator to behave correctly in this case so that I can "undo" the conversion that has happened with my entities at runtime and not flush the wrong value out?
Ultimately the problem was due to the EventTypes of the event listeners added with the EventListenerRegistry. What worked was using EventType.POST_LOAD for all the read operations combined with EventType.PRE_UPDATE and EventType.PRE_INSERT for writes that call a helper method for handling both the same way.
To prevent unneeded writes after making your entity updates it's a good idea to reset the data used for tracking if the entity is dirty in EntityEntry called loadedState. This is a private field in Hibernate 4 so you'll need to use Reflection, however in Hibernate 5 it's available via the getLoadedState() method. One more gotcha is you need to update values of the "state" used when actually flushing the values to the database by the PreInsertEvent and PreUpdateEvent which can be retrieved from the getState() method defined in each.
Related
I am getting a lot of SQL Warning in the logs with the spring boot and sybase.
o.h.engine.jdbc.spi.SqlExceptionHelper : [] SQL Warning Code: 0, SQLState: 010SK
o.h.engine.jdbc.spi.SqlExceptionHelper : [] 010SK: Database cannot set connection option SET_READONLY_TRUE.
o.h.engine.jdbc.spi.SqlExceptionHelper : [] 010SK: Database cannot set connection option SET_READONLY_FALSE.
Could anyone explain the reason behind this?
Solution 1:
java.sql.Connection has a setReadOnly(boolean) method that is meant to notify the database of the type of result set being requested in order to perform any optimizations. However Sybase ASE doesn't require any optimizations, therefore setReadOnly() produces a SQLWarning.
In order to suppress the message you'll need to update the spt_mda table in the MASTER database.
update spt_mda set querytype = 4, set query = '0'
where mdinfo = 'SET_READONLY_FALSE'
and
update spt_mda set querytype = 4, set query = '0'
where mdinfo = 'SET_READONLY_TRUE'
These two entries (they are the only ones) are set to a querytype of 3 by default, which means "not supported", which explains the SQLWarning.
Changing them to a 4 (meaning boolean values) with a query type of "0" basically causes the JDBC Driver to return false without the warning..
Solution 2:
You might turn off/on on logging for specific part of hibernate logging modules, these are different configurations:
# Hibernate logging
# Log everything (a lot of information, but very useful for troubleshooting)
log4j.logger.org.hibernate=FATAL
# Log all SQL DML statements as they are executed
log4j.logger.org.hibernate.SQL=INHERITED
# Log all JDBC parameters
log4j.logger.org.hibernate.type=INHERITED
# Log all SQL DDL statements as they are executed
log4j.logger.org.hibernate.tool.hbm2ddl=INHERITED
# Log the state of all entities (max 20 entities) associated with the session at flush time
log4j.logger.org.hibernate.pretty=INHERITED
# Log all second-level cache activity
log4j.logger.org.hibernate.cache=INHERITED
# Log all OSCache activity - used by Hibernate
log4j.logger.com.opensymphony.oscache=INHERITED
# Log transaction related activity
log4j.logger.org.hibernate.transaction=INHERITED
# Log all JDBC resource acquisition
log4j.logger.org.hibernate.jdbc=INHERITED
# Log all JAAS authorization requests
log4j.logger.org.hibernate.secure=INHERITED
Possible values:
OFF
FATAL
ERROR
WARN
INFO
DEBUG
TRACE
ALL
I have a table where I am inserting records using record.insert() method. I believe this method is doing an insert and then a select but in a different transactions. At the same time I have another thread which pools this table for records processes them and then deletes them.
In some cases I am getting the below exception:
org.jooq.exception.NoDataFoundException: Exactly one row expected for refresh. Record does not exist in database.
at org.jooq.impl.UpdatableRecordImpl.refresh(UpdatableRecordImpl.java:345)
at org.jooq.impl.TableRecordImpl.getReturningIfNeeded(TableRecordImpl.java:232)
at org.jooq.impl.TableRecordImpl.storeInsert0(TableRecordImpl.java:208)
at org.jooq.impl.TableRecordImpl$1.operate(TableRecordImpl.java:169)
My solution was to use DSL.using(configuration()).insertInto instead of record.insert().
My question is shouldn't the insert and fetch be done in the same transaction?
UPDATE:
This is a dropwizard app that is using jooqbundle: com.bendb.dropwizard:dropwizard-jooq.
The configuration is injected in the DAO, the insert is as follows:
R object = // jooq record
object.attach(configuration);
object.insert();
On the second thread I am just selecting some records from this table, processing them and then deleting them
Jooq logs clearly shows that the 2 queries are not run in same transaction:
14:07:09.550 [main] DEBUG org.jooq.tools.LoggerListener - -> with bind values : insert into "queue"
....
14:07:09.083', 1)
14:07:09.589 [main] DEBUG org.jooq.tools.LoggerListener - Affected row(s) : 1
14:07:09.590 [main] DEBUG org.jooq.tools.StopWatch - Query executed : Total: 47.603ms
14:07:09.591 [main] DEBUG org.jooq.tools.StopWatch - Finishing : Total: 48.827ms, +1.223ms
14:07:09.632 [main] DEBUG org.jooq.tools.LoggerListener - Executing query : select "queue"."
I do not see the "autocommit off" or "savepoint" statements in the logs which are generally printed by jooq in case the queries are run in a transaction. I hope this helps, let me know if you need more info
UPDATE 2:
Jooq version is 3.9.1
mysql version 5.6.23
Database and jooq entry yml file:
database:
driverClass: com.mysql.jdbc.Driver
user: ***
password: ***
url: jdbc:mysql://localhost:3306/mySchema
properties:
charSet: UTF-8
characterEncoding: UTF-8
# the maximum amount of time to wait on an empty pool before throwing an exception
maxWaitForConnection: 1s
# the SQL query to run when validating a connection's liveness
validationQuery: "SELECT 1"
# the timeout before a connection validation queries fail
validationQueryTimeout: 3s
# initial number of connections
initialSize: 25
# the minimum number of connections to keep open
minSize: 25
# the maximum number of connections to keep open
maxSize: 25
# whether or not idle connections should be validated
checkConnectionWhileIdle: true
# the amount of time to sleep between runs of the idle connection validation, abandoned cleaner and idle pool resizing
evictionInterval: 10s
# the minimum amount of time an connection must sit idle in the pool before it is eligible for eviction
minIdleTime: 1 minute
jooq:
# The flavor of SQL to generate. If not specified, it will be inferred from the JDBC connection URL. (default: null)
dialect: MYSQL
# Whether to write generated SQL to a logger before execution. (default: no)
logExecutedSql: no
# Whether to include schema names in generated SQL. (default: yes)
renderSchema: yes
# How names should be rendered in generated SQL. One of QUOTED, AS_IS, LOWER, or UPPER. (default: QUOTED)
renderNameStyle: QUOTED
# How keywords should be rendered in generated SQL. One of LOWER, UPPER. (default: UPPER)
renderKeywordStyle: UPPER
# Whether generated SQL should be pretty-printed. (default: no)
renderFormatted: no
# How parameters should be represented. One of INDEXED, NAMED, or INLINE. (default: INDEXED)
paramType: INDEXED
# How statements should be generated; one of PREPARED_STATEMENT or STATIC_STATEMENT. (default: PREPARED_STATEMENT)
statementType: PREPARED_STATEMENT
# Whether internal jOOQ logging should be enabled. (default: no)
executeLogging: no
# Whether optimistic locking should be enabled. (default: no)
executeWithOptimisticLocking: yes
# Whether returned records should be 'attached' to the jOOQ context. (default: yes)
attachRecords: yes
# Whether primary-key fields should be updatable. (default: no)
updatablePrimaryKeys: no
Have included the jooq bundle in the Application class as described in https://github.com/benjamin-bader/droptools/tree/master/dropwizard-jooq.
Using https://github.com/xvik/dropwizard-guicey to inject the configuration into each DAO.
The guide module has the following binding:
bind(Configuration.class).toInstance(jooqBundle.getConfiguration());
I'm trying to implement some kind of semaphores for cron jobs that runs in different JBoss nodes. I'm trying to use the database (Oracle 11g) as a locking mechanism using one table to syncronize the cron jobs in the different nodes.
The table is very simple:
CREATE TABLE SYNCHRONIZED_CRON_JOB_TASK
(
ID NUMBER(10) NOT NULL,
CRONJOBTYPE VARCHAR2(255 Byte),
CREATIONDATE TIMESTAMP(6) NOT NULL,
RUNNING NUMBER(1)
);
ALTER TABLE SYNCHRONIZED_CRON_JOB_TASK
ADD CONSTRAINT PK_SYNCHRONIZED_CRON_JOB_TASK
PRIMARY KEY (ID);
So when a job starts it searches in the table for a entry of its cronjobtype, and checks if it is already running. If not it updates the entry setting running flag to true. This first select is made with JPA CriteriaApi using Hibernate and Pessimistic Lock.
query.setLockMode(javax.persistence.LockModeType.PESSIMISTIC_WRITE);
All those opperations are made within one transaction.
When one process runs, the querys it makes are the following:
[Server:server-two] 10:38:00,049 INFO [stdout] (scheduler-2) 2015-04-30 10:38:00,048 WARN (Loader.java:264) - HHH000444: Encountered request for locking however dialect reports that database prefers locking be done in a separate select (follow-on locking); results will be locked after initial query executes
[Server:server-two] 10:38:00,049 INFO [stdout] (scheduler-2) Hibernate: select * from ( select distinct synchroniz0_.id as id1_127_, synchroniz0_.creationDate as creation2_127_, synchroniz0_.running as running3_127_, synchroniz0_.CRONJOBTYPE as CRONJOBT4_127_ from SYNCHRONIZED_CRON_JOB_TASK synchroniz0_ where synchroniz0_.CRONJOBTYPE=? ) where rownum <= ?
[Server:server-two] 10:38:00,053 INFO [stdout] (scheduler-2) Hibernate: select id from SYNCHRONIZED_CRON_JOB_TASK where id =? for update
[Server:server-two] 10:38:00,056 INFO [stdout] (scheduler-2) Hibernate: update SYNCHRONIZED_CRON_JOB_TASK set creationDate=?, running=?, CRONJOBTYPE=? where id=?
There is no problem with this warning, you can see a first select and then a select for update, so Oracle should block other select operations on this row.
But that's the point, the queries are not being blocked so two jobs can enter and make the select and update without problem. The lock is not working, we can see it if we run two cron jobs simultaneously:
[Server:server-one] 10:38:00,008 INFO [stdout] (scheduler-3) 2015-04-30 10:38:00,008 WARN (Loader.java:264) - HHH000444: Encountered request for locking however dialect reports that database prefers locking be done in a separate select (follow-on locking); results will be locked after initial query executes
[Server:server-two] 10:38:00,008 INFO [stdout] (scheduler-2) 2015-04-30 10:38:00,008 WARN (Loader.java:264) - HHH000444: Encountered request for locking however dialect reports that database prefers locking be done in a separate select (follow-on locking); results will be locked after initial query executes
[Server:server-two] 10:38:00,009 INFO [stdout] (scheduler-2) Hibernate: select * from ( select distinct synchroniz0_.id as id1_127_, synchroniz0_.creationDate as creation2_127_, synchroniz0_.running as running3_127_, synchroniz0_.CRONJOBTYPE as CRONJOBT4_127_ from SYNCHRONIZED_CRON_JOB_TASK synchroniz0_ where synchroniz0_.CRONJOBTYPE=? ) where rownum <= ?
[Server:server-one] 10:38:00,009 INFO [stdout] (scheduler-3) Hibernate: select * from ( select distinct synchroniz0_.id as id1_127_, synchroniz0_.creationDate as creation2_127_, synchroniz0_.running as running3_127_, synchroniz0_.CRONJOBTYPE as CRONJOBT4_127_ from SYNCHRONIZED_CRON_JOB_TASK synchroniz0_ where synchroniz0_.CRONJOBTYPE=? ) where rownum <= ?
[Server:server-two] 10:38:00,013 INFO [stdout] (scheduler-2) Hibernate: select id from SYNCHRONIZED_CRON_JOB_TASK where id =? for update
[Server:server-one] 10:38:00,014 INFO [stdout] (scheduler-3) Hibernate: select id from SYNCHRONIZED_CRON_JOB_TASK where id =? for update
[Server:server-two] 10:38:00,016 INFO [stdout] (scheduler-2) 2015-04-30 10:38:00,015 DEBUG (SynchronizedCronJobService.java:65) - Task read SynchronizedCronJobTask [id=185, type=AlertMailTaskExecutor, creationDate=2015-04-25 07:11:33.0, running=false]
[Server:server-two] 10:38:00,018 INFO [stdout] (scheduler-2) Hibernate: update SYNCHRONIZED_CRON_JOB_TASK set creationDate=?, running=?, CRONJOBTYPE=? where id=?
[Server:server-one] 10:38:00,022 INFO [stdout] (scheduler-3) 2015-04-30 10:38:00,022 DEBUG (SynchronizedCronJobService.java:65) - Task read SynchronizedCronJobTask [id=185, type=AlertMailTaskExecutor, creationDate=2015-04-25 07:11:33.0, running=false]
[Server:server-one] 10:38:00,024 INFO [stdout] (scheduler-3) Hibernate: update SYNCHRONIZED_CRON_JOB_TASK set creationDate=?, running=?, CRONJOBTYPE=? where id=?
I've tried to make this select for update on a SQL tool (SQLWorkbenchJ) with two connections and the bloking is working fine within this tool. But if i make this select for update on the SQL tool and launch the cron jobs, they are not bloked and run without problems.
I think the problem comes from JPA, Hibernate or the Oracle driver but i'm not sure. Any idea on where is the problem? Should i use anotehr strategy?
Thanks in advance.
Finally i managed to make it work but with some modiffications. The idea is to use LockModeType.PESSIMISTIC_FORCE_INCREMENT instead of PESSIMISTIC_WRITE. Using this lock mode the Cron Jobs behave as follows:
When the first job makes the select for update everything goes as expected but the version on the object changes.
If another job tries to make the same select while the first is still on its transaction, JPA launches a OptimisticLockException so if you catch that exception you can be sure that it was thrown for a read lock.
This solution has various counterparts:
SynchronizedCronJobTask must have a version field and be under version control with #Version
You need to handle OptimisticLockException, and it should be catch outside the transactional service method in order to make rollback when de lock happens.
IMHO is a non elegant solution, much worse than simply a lock where the Cron Jobs wait for the previous Jobs to finish.
I can confirm Ricardos observation. I have several Lock-Modes tested with a H2-Database, and all worked as expected. Neither of the pessimistic Lock-Modes worked correctly in combination with an Oracle database. I did not try optimistic locking, but it's amazing that there's a lockmode that doesn't work with the top dog at all.
Set locking mode to PESSIMISTIC_READ, because you need that second server knew about changes of first server before changing data.
I'm using PostgreSQL 8.2.9, Solr 3.1, Tomcat 5.5
I have following problem:
When I execute delta-import - /dataimport?command=delta-import - any update queries to database are not responding for about 30 seconds.
I can easily repeat this behaviour (using psql or hibernate):
PSQL:
Execute delta-import
Immediately in psql - run SQL query: 'UPDATE table SET ... WHERE id = 1;' several times
The second/third/... time - I must wait ~30 seconds for query to return
Hibernate:
In logs - hibernate waits ~30 seconds on method 'doExecuteBatch(...)' after setting query parameters
No other queries are executed when I'm testing this. On the other hand when I'm executing other commands (like full-import, etc.)- everything works perfectly fine.
In Solr's dataconfig.xml:
I have readOnly attribute set to true on PostgreSQL dataSource.
deltaImportQuery, deltaQuery, ... on entity tags don't lock database (simple SELECT's)
Web app (using hibernate) logs:
2012-01-08 18:54:52,403 DEBUG my.package.Class1... Executing method: save
2012-01-08 18:55:26,707 DEBUG my.package.Class1... Executing method: list
Solr logs:
INFO: [] webapp=/search path=/dataimport params={debug=query&command=delta-import&commit=true} status=0 QTime=1
2012-01-08 18:54:50 org.apache.solr.handler.dataimport.DataImporter doDeltaImport
INFO: Starting Delta Import
...
2012-01-08 18:54:50 org.apache.solr.handler.dataimport.JdbcDataSource$1 call
INFO: Time taken for getConnection(): 4
...
FINE: Executing SQL: select ... where last_edit_date > '2012-01-08 18:51:43'
2012-01-08 18:54:50 org.apache.solr.core.Config getNode
...
FINEST: Time taken for sql :4
...
INFO: Import completed successfully
2012-01-08 18:54:50 org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit(optimize=true,waitFlush=false,waitSearcher=true,expungeDeletes=false)
2012-01-08 18:54:53 org.apache.solr.core.Config getNode
...
2012-01-08 18:54:53 org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
...
2012-01-08 18:54:53 org.apache.solr.handler.dataimport.DocBuilder execute
INFO: Time taken = 0:0:2.985
There're no 'SELECT ... FOR UPDATE / LOCK / etc.' queries in above logs.
I have set logging for PostgreSQL - there're no locks. Even sessions are set to:
Jan 11 14:33:07 test postgres[26201]: [3-1] <26201> LOG: execute <unnamed>: SET SESSION CHARACTERISTICS AS TRANSACTION READ ONLY
Jan 11 14:33:07 test postgres[26201]: [4-1] <26201> LOG: execute <unnamed>: SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL READ UNCOMMITTED
Why is this happening? This looks like some kind of database lock but then why when import is completed (2 secs) queries are still waiting (for 30 secs)?
The Update waiting for the SELECT statement to complete before executing. Not a lot you can do about that that I'm aware of. We get around the issue by doing our indexing in batches. Multiple SELECT statements are fine, but UPDATE and DELETE affect the records and wont execute until it can lock the table.
Ok. It was hard to find solution to this problem.
The reason was the underlying platform - disk write saturation. There was too many small "disk-write"'s that consumed all disk-write power.
Now we have new agreement with our service layer provider.
Test query:
while true ; do echo "UPDATE table_name SET type='P' WHERE type='P'" | psql -U user -d database_name ; sleep 1 ; done
plus making changes via out other application, plus update index simultaneously.
This was before platform change:
And here is how it works now:
This program does tens of thousands of consecutive inserts one after the other. I've never used Hibernate before. I'm getting extremely slow performance (if I just connect and execute the SQL manually I am 10-12x quicker. My batch_size is set to 50 as per many hibernate tutorials.
Here is a log from a single insert - perhaps you could help me understand exactly what is happening:
START INSERT
11:02:56.121 [main] DEBUG org.hibernate.impl.SessionImpl - opened session at timestamp: 13106053761
11:02:56.121 [main] DEBUG o.h.transaction.JDBCTransaction - begin
11:02:56.121 [main] DEBUG org.hibernate.jdbc.ConnectionManager - opening JDBC connection
11:02:56.121 [main] TRACE o.h.c.DriverManagerConnectionProvider - total checked-out connections: 0
11:02:56.121 [main] TRACE o.h.c.DriverManagerConnectionProvider - using pooled JDBC connection, pool size: 0
11:02:56.121 [main] DEBUG o.h.transaction.JDBCTransaction - current autocommit status: false
11:02:56.121 [main] TRACE org.hibernate.jdbc.JDBCContext - after transaction begin
11:02:56.121 [main] TRACE org.hibernate.impl.SessionImpl - setting flush mode to: MANUAL
11:02:56.121 [main] TRACE o.h.e.def.DefaultLoadEventListener - loading entity: [com.xyzcompany.foo.edoi.ejb.msw000.MSW000Rec#component[keyW000]{keyW000=F000 ADSUFC}]
11:02:56.121 [main] TRACE o.h.e.def.DefaultLoadEventListener - creating new proxy for entity
11:02:56.122 [main] TRACE o.h.e.d.DefaultSaveOrUpdateEventListener - saving transient instance
11:02:56.122 [main] DEBUG o.h.e.def.AbstractSaveEventListener - generated identifier: component[keyW000]{keyW000=F000 ADSUFC}, using strategy: org.hibernate.id.CompositeNestedGeneratedValueGenerator
11:02:56.122 [main] TRACE o.h.e.def.AbstractSaveEventListener - saving [com.xyzcompany.foo.edoi.ejb.msw000.MSW000Rec#component[keyW000]{keyW000=F000 ADSUFC}]
11:02:56.123 [main] TRACE o.h.e.d.AbstractFlushingEventListener - flushing session
11:02:56.123 [main] DEBUG o.h.e.d.AbstractFlushingEventListener - processing flush-time cascades
11:02:56.123 [main] DEBUG o.h.e.d.AbstractFlushingEventListener - dirty checking collections
11:02:56.123 [main] TRACE o.h.e.d.AbstractFlushingEventListener - Flushing entities and processing referenced collections
11:02:56.125 [main] TRACE o.h.e.d.AbstractFlushingEventListener - Processing unreferenced collections
11:02:56.125 [main] TRACE o.h.e.d.AbstractFlushingEventListener - Scheduling collection removes/(re)creates/updates
11:02:56.126 [main] DEBUG o.h.e.d.AbstractFlushingEventListener - Flushed: 1 insertions, 0 updates, 0 deletions to 62 objects
11:02:56.126 [main] DEBUG o.h.e.d.AbstractFlushingEventListener - Flushed: 0 (re)creations, 0 updates, 0 removals to 0 collections
11:02:56.132 [main] TRACE o.h.e.d.AbstractFlushingEventListener - executing flush
11:02:56.132 [main] TRACE org.hibernate.jdbc.ConnectionManager - registering flush begin
11:02:56.132 [main] TRACE o.h.p.entity.AbstractEntityPersister - Inserting entity: [com.xyzcompany.foo.edoi.ejb.msw000.MSW000Rec#component[keyW000]{keyW000=F000 ADSUFC}]
11:02:56.132 [main] DEBUG org.hibernate.jdbc.AbstractBatcher - about to open PreparedStatement (open PreparedStatements: 0, globally: 0)
11:02:56.132 [main] DEBUG org.hibernate.SQL - insert into MSW000 (W000_DATA_REC, W000_FILE_FLAGS, KEY_W000) values (?, ?, ?)
11:02:56.132 [main] TRACE org.hibernate.jdbc.AbstractBatcher - preparing statement
11:02:56.132 [main] TRACE o.h.p.entity.AbstractEntityPersister - Dehydrating entity: [com.xyzcompany.foo.edoi.ejb.msw000.MSW000Rec#component[keyW000]{keyW000=F000 ADSUFC}]
11:02:56.132 [main] TRACE org.hibernate.type.StringType - binding ' ADSUFCA ' to parameter: 1
11:02:56.132 [main] TRACE org.hibernate.type.StringType - binding ' ' to parameter: 2
11:02:56.132 [main] TRACE org.hibernate.type.StringType - binding 'F000 ADSUFC' to parameter: 3
11:02:56.132 [main] DEBUG org.hibernate.jdbc.AbstractBatcher - Executing batch size: 1
11:02:56.133 [main] DEBUG org.hibernate.jdbc.AbstractBatcher - about to close PreparedStatement (open PreparedStatements: 1, globally: 1)
11:02:56.133 [main] TRACE org.hibernate.jdbc.AbstractBatcher - closing statement
11:02:56.133 [main] TRACE org.hibernate.jdbc.ConnectionManager - registering flush end
11:02:56.133 [main] TRACE o.h.e.d.AbstractFlushingEventListener - post flush
11:02:56.133 [main] DEBUG o.h.transaction.JDBCTransaction - commit
11:02:56.133 [main] TRACE org.hibernate.impl.SessionImpl - automatically flushing session
11:02:56.133 [main] TRACE org.hibernate.jdbc.JDBCContext - before transaction completion
11:02:56.133 [main] TRACE org.hibernate.impl.SessionImpl - before transaction completion
11:02:56.133 [main] DEBUG o.h.transaction.JDBCTransaction - committed JDBC Connection
11:02:56.133 [main] TRACE org.hibernate.jdbc.JDBCContext - after transaction completion
11:02:56.133 [main] DEBUG org.hibernate.jdbc.ConnectionManager - transaction completed on session with on_close connection release mode; be sure to close the session to release JDBC resources!
11:02:56.133 [main] TRACE org.hibernate.impl.SessionImpl - after transaction completion
11:02:56.133 [main] TRACE org.hibernate.impl.SessionImpl - closing session
11:02:56.133 [main] TRACE org.hibernate.jdbc.ConnectionManager - performing cleanup
11:02:56.133 [main] DEBUG org.hibernate.jdbc.ConnectionManager - releasing JDBC connection [ (open PreparedStatements: 0, globally: 0) (open ResultSets: 0, globally: 0)]
11:02:56.133 [main] TRACE o.h.c.DriverManagerConnectionProvider - returning connection to pool, pool size: 1
11:02:56.133 [main] TRACE org.hibernate.jdbc.JDBCContext - after transaction completion
11:02:56.133 [main] DEBUG org.hibernate.jdbc.ConnectionManager - transaction completed on session with on_close connection release mode; be sure to close the session to release JDBC resources!
11:02:56.134 [main] TRACE org.hibernate.impl.SessionImpl - after transaction completion
FINISH INSERT
When you call session.save(), hibernate will generate an INSERT SQL. This INSERT SQL will be appended to be issued to the DB during flushing (i.e session.flush()) .
During flushing, if hibernate.jdbc.batch_size is set to some non-zero value, Hibernate will use the batching feature introduced in the JDBC2 API to issue the batch insert SQL to the DB .
For example , if you save() 100 records and your hibernate.jdbc.batch_size is set to 50. During flushing, instead of issue the following SQL 100 times :
insert into TableA (id , fields) values (1, 'val1');
insert into TableA (id , fields) values (2, 'val2');
insert into TableA (id , fields) values (3, 'val3');
.........................
insert into TableA (id , fields) values (100, 'val100');
Hiberate will group them in batches of 50 , and only issue 2 SQL to the DB, like this:
insert into TableA (id , fields) values (1, 'val1') , (2, 'val2') ,(3, 'val3') ,(4, 'val4') ,......,(50, 'val50')
insert into TableA (id , fields) values (51, 'val51') , (52, 'val52') ,(53, 'val53') ,(54, 'val54'),...... ,(100, 'val100')
Please note that Hibernate would disable insert batching at the JDBC level transparently if the primary key of the inserting table isGenerationType.Identity.
From your log: you save() only one record and then flush(), so there is only one appending INSERT SQL to be processed for every flush. That's why Hibernate cannot help you to batch inserting as there is only one INSERT SQL to be processed. You should save() up to the certain amount of records before calling flush() instead of calling flush() for every save().
The best practise of batch inserting is something like this:
Session session = sessionFactory.openSession();
Transaction tx = session.beginTransaction();
for ( int i=0; i<888888; i++ ) {
TableA record = new TableA();
record.setXXXX();
session.save(record)
if ( i % 50 == 0 ) { //50, same as the JDBC batch size
//flush a batch of inserts and release memory:
session.flush();
session.clear();
}
}
tx.commit();
session.close();
You save and flush the records batch by batch. In the end of each batch you should clear the persistence context to release some memory to prevent memory exhaustion as every persistent object is placed into the first level cache (your JVM's memory). You could also disable the second-level cache to reduce the unnecessary overhead.
Reference:
Official Hibernate Documentation : Chapter 14. Batch processing
Hibernate Batch Processing – Why you may not be using it. (Even if you think you are)
If you must use hibernate for huge batch jobs StatelessSession is the way to go. It strips things down to the most basic converting-objects-to-SQL-statements mapping and eliminates all of the overhead of the ORM features you're not using when just cramming rows into the DB wholesale.
It would also be much easier to make suggestions on your actual code than the log :)
11:02:56.133 [main] DEBUG o.h.transaction.JDBCTransaction - commit
This is saying that the database is committing after every insert. Ensure you are not committing your transaction / closing your session inside the insert loop. Do this once at the end instead.