I am using Hibernate 5.2.8.Final version and we have a requirement where we read millions of data from database and update the data by some business logic, as my database is Huge I want to commit data after my batchsize is reached so I have written below code
Session session = HibernateUtil.getSessionFactory().getCurrentSession();
session.getTransaction().begin();
Query<Object> query = session.createQuery(SQL, Object.class);
ScrollableResults scrollableResults = query.setCacheMode(CacheMode.IGNORE)
.scroll(ScrollMode.FORWARD_ONLY);
int count = 0;
while (scrollableResults.next())
{
Object object = (Object) scrollableResults.get(0);
process(object)
session.update(object);
if (++count % batchSizeDeclare== 0)
{
session.flush();
session.clear();
LOGGER.info("Updated batch records");
}
}
session.getTransaction().commit();
LOGGER.info("commited in Hibernate ");
}
Below is my hibernate.cfg.xml file
<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD//EN"
"http://www.hibernate.org/dtd/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<!-- Database connection settings -->
<property name="connection.driver_class">com.microsoft.sqlserver.jdbc.SQLServerDriver</property>
<property name="connection.url">jdbc:sqlserver://com;database=DEV</property>
<property name="connection.username">user</property>
<property name="connection.password">pass</property>
<property name="hibernate.default_schema">schema</property>
<!-- JDBC connection pool (use the built-in) -->
<property name="connection.pool_size">5</property>
<!-- SQL dialect -->
<property name="dialect">org.hibernate.dialect.SQLServer2012Dialect</property>
<!-- Disable the second-level cache -->
<property name="cache.provider_class">org.hibernate.cache.internal.NoCacheProvider</property>
<!-- Echo all executed SQL to stdout -->
<!-- <property name="show_sql">true</property> -->
<!-- <property name="format_sql">true</property> -->
<!-- Drop and re-create the database schema on startup -->
<property name="hbm2ddl.auto">update</property>
<!-- org.hibernate.HibernateException: No CurrentSessionContext configured! -->
<property name="hibernate.current_session_context_class">thread</property>
<property name="hibernate.jdbc.batch_size">100</property>
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">20</property>
<property name="hibernate.c3p0.timeout">300</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.idle_test_period">3000</property>
<mapping class="com.beans.Object" />
</session-factory>
</hibernate-configuration>
Below is my Object.java
public class Object implements Serializable
{
private static final long serialVersionUID = 1L;
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name = "ID", nullable = false, unique = true, updatable = false)
private int id;
private String filePath;
private String fileName;
private String mimeType;
private double fileSize;
// removed getters and setters
}
Once my code reached session.flush() it is not doing any thing even after waiting for 30 min. Is this the correct way to batch commit? How to batch Update?
Once my code reached session.flush it is not doing anything even
after waiting for 30 min.
On the contrary, the database seems to be doing way too much. It's just that you don't see any progress because the database is struggling to cope with the huge amount of work that you submitted.
Is this the correct way to batch commit?
The short answer is No.
You don't have to fetch millions of rows from the DB. You have better options:
You can do the processing in the database, so that you don't pay the price of extracting data and sending it over the network, only to process it in Java.
If you can't process it in the DB, then you need to use a batch processor that only fetches small chunks of data at a time. This way, you can even parallelize the batch processing, which should reduce the overall processing time.
Related
I am trying to figure out the different types of generating primary keys for classes that inherit from a superclass, I am using the embedded H2 database, the data is stored in a file. Everything worked well, the entities that I had previously created manually in the database were loaded, but when I tried to save a new entity using Hibernate, I got an error
Table "ID_GENERATOR" not found.
I wonder, should I create the table for "ID_GENERATOR" table generator manually? I thought Hibernate creates it automatically
Here' my superclass:
#Entity
#Inheritance(strategy = InheritanceType.TABLE_PER_CLASS)
public class FundEvent implements Comparable<FundEvent>{
#Id
#TableGenerator(name = "id_generator")
#GeneratedValue(strategy = GenerationType.TABLE, generator = "id_generator")
protected int id;
#Column(name = "amount")
protected int amount;
#Column(name = "deadline")
protected Date deadline;
#ManyToOne
#JoinColumn(name = "user_id")
protected User user;
//getters and setters
And here's Entity class object of which I'm trying to save:
#Entity
#Table(name = "purchases")
public class Purchase extends FundEvent{
#Column(name = "goal")
private String goal;
#Column(name = "bought")
private boolean bought;
#Column(name = "removed")
private boolean removed;
#Transient
private int activeAmount;
//getters and setters
Do I miss something?
UPD: Here's my hibernate.cfg.xml. Note, i'm not using Spring, just Hibernate
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD 3.0//EN"
"http://www.hibernate.org/dtd/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<!-- JDBC Database connection settings -->
<property name="connection.driver_class">org.h2.Driver</property>
<property name="connection.url">jdbc:h2:file:/home/vetal22331122/data_for_purch</property>
<property name="connection.username"></property>
<property name="connection.password"></property>
<!-- JDBC connection pool settings ... using built-in test pool -->
<property name="connection.pool_size">2</property>
<!-- Select our SQL dialect -->
<property name="dialect">org.hibernate.dialect.H2Dialect</property>
<!-- Echo the SQL to stdout -->
<property name="show_sql">true</property>
<!-- Set the current session context -->
<property name="current_session_context_class">thread</property>
<!-- Drop and re-create the database schema on startup -->
<property name="hbm2ddl.auto">none</property>
<!-- dbcp connection pool configuration -->
<property name="hibernate.dbcp.initialSize">5</property>
<property name="hibernate.dbcp.maxTotal">20</property>
<property name="hibernate.dbcp.maxIdle">10</property>
<property name="hibernate.dbcp.minIdle">5</property>
<property name="hibernate.dbcp.maxWaitMillis">-1</property>
<mapping class="entities.Purchase"/>
<!--<mapping class="entities.Payment"/>-->
<mapping class="entities.User"/>
</session-factory>
</hibernate-configuration>
Javadoc of TableGenerator showing an example of it filled with almost all the properties, try to fill in every property of the annotation and run again: https://docs.oracle.com/javaee/6/api/javax/persistence/TableGenerator.html
First, let me list the things I have used:
Eclipse JEE version 2021-03
Apache Tomcat Server 9
Hibernate ORM version 5.2.18.Final
PostgreSQL 14
Java 8
Some driver I have used: the required in lib of Hibernate ORM, postgresql-42.2.22.jar, jaxb-api-1.0.jar
Second is my code:
In the main class, I use it to run the application I let the name of class is CreateStudentDemo in the phucldh.Demo package in the src folder
public static void main(String[] args) {
// create session factory
SessionFactory factory = new Configuration().configure("hibernate.cfg.xml").addAnnotatedClass(Student.class).buildSessionFactory();
// create session
Session session = factory.getCurrentSession();
try {
// create a student object
Student tempStudent = new Student("Le", "Phuc", "phucldh.work#gmail.com");
// start a transaction
session.beginTransaction();
// save the student object
session.save(tempStudent);
// commit transaction
session.getTransaction().commit();
} catch (Exception e) {
System.out.println("Create student demo error: " + e.getMessage());
} finally {
factory.close();
}
}
And to connect to PostgreSQL I have a configuration file hibernate.cfg.xml in the src folder and the content of this file:
<session-factory>
<!-- JDBC Database connection settings -->
<property name="connection.driver_class">org.postgresql.Driver</property>
<property name="connection.url">jdbc:postgresql://localhost:5432/HibernateLearn</property>
<property name="connection.username">postgres</property>
<property name="connection.password">********</property>
<!-- JDBC connection pool settings ... using built-in test pool -->
<property name="connection.pool_size">1</property>
<!-- Select our SQL dialect -->
<property name="hibernate.dialect">org.hibernate.dialect.PostgreSQLDialect</property>
<!-- Set the current session context -->
<property name="current_session_context_class">thread</property>
<!-- Disable the second-level cache -->
<property name="cache.provider_class">org.hibernate.cache.internal.NoCacheProvider</property>
<!-- Echo the SQL to stdout -->
<property name="show_sql">true</property>
<!-- Drop and re-create the database schema on startup -->
<property name="hbm2ddl.auto">create</property>
</session-factory>
That all I have done but when I running I have a problem:
INFO: HHH000206: hibernate.properties not found
Exception in thread "main" java.lang.NoSuchMethodError: 'javax.xml.bind.JAXBContext javax.xml.bind.JAXBContext.newInstance(java.lang.Class[])'
at org.hibernate.boot.cfgxml.internal.JaxbCfgProcessor.unmarshal(JaxbCfgProcessor.java:122)
at org.hibernate.boot.cfgxml.internal.JaxbCfgProcessor.unmarshal(JaxbCfgProcessor.java:65)
at org.hibernate.boot.cfgxml.internal.ConfigLoader.loadConfigXmlResource(ConfigLoader.java:57)
at org.hibernate.boot.registry.StandardServiceRegistryBuilder.configure(StandardServiceRegistryBuilder.java:163)
at org.hibernate.cfg.Configuration.configure(Configuration.java:258)
at phucldh.Demo.CreateStudentDemo.main(CreateStudentDemo.java:15)
And I see that line 15 of CreateStudentDemo.java is the line about
SessionFactory factory = new Configuration().configure("hibernate.cfg.xml").addAnnotatedClass(Student.class).buildSessionFactory();
So I hope that anybody can help me find what I have wrong. Thank everybody very much. Hope all have a nice day.
So i got this project where i m storing a "Categorie" class in data base with onetomany relationship with "SousCategorie" class which works fine and it creates a table named "categorie_sous_categories"
when i tired to retrieve the data from categorie_sous_categories table an error occured that this class is mapped which hibernate had created the table bu i didin't declare the class in my program
my program structure:
the exception log:
Exception in thread "main" java.lang.IllegalArgumentException: org.hibernate.hql.internal.ast.QuerySyntaxException: Categorie_sous_categories is not mapped [select categorie_matricule from categorie_sous_categories]
the code i try to run:
SessionFactory sessionFactory = new Configuration().
configure().buildSessionFactory();
Session session = sessionFactory.openSession();
session.beginTransaction();
List<String> categorieList = session.createQuery("select categorie_matricule from categorie_sous_categories").list();
for (String produit:categorieList){
System.out.println(produit);
}
session.getTransaction().commit();
session.close();
the table i try to retrive data from:
my hibernate.cfg.xml file:
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD 3.0//EN"
"http://www.hibernate.org/dtd/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="hibernate.connection.url">jdbc:postgresql://localhost:5432/DevIT</property> <!-- BD Mane -->
<property name="connection.driver_class">org.postgresql.Driver</property> <!-- DB Driver -->
<property name="connection.username">postgres</property> <!-- DB User -->
<property name="connection.password">test123</property> <!-- DB Password -->
<property name="dialect">org.hibernate.dialect.PostgreSQLDialect</property> <!-- DB Dialect -->
<property name="hbm2ddl.auto">update</property> <!-- create / create-drop / update -->
<property name="show_sql">true</property> <!-- Show SQL in console -->
<property name="format_sql">true</property> <!-- Show SQL formatted -->
<property name="hibernate.current_session_context_class">thread</property>
<mapping class="CoreApp.Categorie"/>
<mapping class="CoreApp.SousCategorie"/>
<mapping class="CoreApp.Produit"/>
</session-factory>
</hibernate-configuration>
try select categorie_matricule from Categorie, it will return the expected result.
Also, exception is self explanatory, table categorie_sous_categories is not mapped with any Entity/Class , it's a mapping table. And it shouldn't be fetched directly, any query you want to run on this table, can be changed - where original classes Categorie and SousCategorie are being used.
I face following issue while trying to write some JBehave BDD test. Scenario is the following:
Due to an MQ message some specific records should be saved into my database and some should be deleted from it. In my #When step I send the given message and in my #Then step I have some asserts to controll the result.
I face issue with the persist and update methods of Hibernate JPA, because it happens just after my code runs to my #Then step and so I got always false result. When I check it in debug mode, and check every recorsd which should be updated / deteled, they are fine.
I assume my asserts should be executed just after the transaction is committed into the database - but this is not the case as I can see.
Is there a way to set a delay or sleep time between database transactions?
Here is e.g. my delete method in my main project:
public void deleteByAbcId(final String Id) {
getEm().createNamedQuery(TABLE.NAMED_QUERY_DELETE_BY_ABC_ID)
.setParameter(Table.QUERY_PARAM_ABC_ID, Id)
.executeUpdate();
}
And in my BDD project I set up database connection as follows:
public class DatabaseService implements Closeable {
private EntityManagerFactory emf = null;
private EntityManager em = null;
/**
* This creates an entity manager based on the db connection parameters received in the argument
*/
public DatabaseService(Properties configuration) {
emf = Persistence.createEntityManagerFactory("project-pu", configuration);
em = emf.createEntityManager();
em.getTransaction().begin();
}
/**
* Returns the entity manager for the db connection
*/
public EntityManager getEm() {
return em;
}
In my assertion I use the following query to check if the given record was successfully removed from the db:
assertNull(dbHelper.findTableIdBasedOnAbcId(Serenity.sessionVariableCalled(ABC_ID)));
My dbHelper class looks like as following:
public class DbHelper {
private DatabaseService database;
private Configuration config = Configuration.getInstance();
public DbHelper() {
database = new DatabaseService(config.getDbProperties());
}
public String findTableIdBasedOnAbcId(String Id) throws Exception {
String query = "SELECT id FROM TABLE WHERE ABC_ID = ?1";
Query queryResult = database.getEm().createNativeQuery(query);
queryResult.setParameter(1, Id);
List<Long> list = (List<Long>) queryResult.getResultList();
if (!list.isEmpty()) {
return String.valueOf(list.get(0));
}
return null;
}
It always retunr the given record of the database, which gets not deleted.
This is the persistence xml of my main project:
<persistence-unit name="aaa-pu" transaction-type="JTA">
<provider>org.hibernate.jpa.HibernatePersistenceProvider</provider>
<jta-data-source>jdbc/aaaa-ds</jta-data-source>
<mapping-file>META-INF/orm.xml</mapping-file>
<properties>
<property name="hibernate.dialect" value="org.hibernate.dialect.Oracle10gDialect"/>
<property name="hibernate.show_sql" value="true"/>
<property name="hibernate.format_sql" value="true"/>
<property name="hibernate.transaction.jta.platform" value="org.hibernate.engine.transaction.jta.platform.internal.WeblogicJtaPlatform"/>
<property name="hibernate.hbm2ddl.auto" value="none"/>
<property name="hibernate.id.new_generator_mappings" value="false"/>
<property name="tomee.jpa.factory.lazy" value="true"/>
</properties>
</persistence-unit>
And this is the one of my BDD project:
<persistence-unit name="project-pu" transaction-type="RESOURCE_LOCAL">
<provider>org.hibernate.jpa.HibernatePersistenceProvider</provider>
<properties>
<property name="hibernate.dialect" value="org.hibernate.dialect.Oracle10gDialect" />
<property name="hibernate.connection.autocommit" value="false" />
<property name="hibernate.hbm2ddl.auto" value="none" />
<property name="hibernate.show_sql" value="false" />
<property name="hibernate.format_sql" value="true" />
<property name="hibernate.generate_statistics" value="false"/>
</properties>
</persistence-unit>
Any idea, how to set transactions not to execute the assert method before the executeUpdate of Hibernate runs?
I tried to call the entityManager.getTransaction.commit method and then start a new one every time but no change.
Is there a way to lock the transaction until the executeUpdate is done and just after that allowing the select query? Or is it possible with JBehave to give a waiting or sleep time between steps?
Thank you.
I have Hsqldb 1) in memory
<jdbc:embedded-database id="idtSimulatorDataSource" type="HSQL">
<jdbc:script location="classpath:hsql/idt_schema.sql"/>
</jdbc:embedded-database>
2) on server-mode
<bean id="idtSimulatorDataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="org.hsqldb.jdbc.JDBCDriver"/>
<property name="username" value="sa"/>
<property name="password" value=""/>
<property name="url" value="jdbc:hsqldb:hsql://localhost:9001/idt_simulatordb"/>
</bean>
1) all works properly
2) can't create schema. If I create schema manually, then hibernate create all necessary DDL manipulations and everethyng works properly too.
But I don't know where should I assign property for db initialization script
Here are my persistence file
<?xml version="1.0" encoding="UTF-8"?>
<persistence xmlns="http://java.sun.com/xml/ns/persistence" version="2.0">
<persistence-unit name="IDTSimulator" transaction-type="RESOURCE_LOCAL">
<provider>org.hibernate.ejb.HibernatePersistence</provider>
<class>com.nxsystems.simulator.idt.domain.IDTAccount</class>
<class>com.nxsystems.simulator.idt.domain.IDTCard</class>
<class>com.nxsystems.simulator.idt.domain.IDTCurrency</class>
<class>com.nxsystems.simulator.idt.domain.IDTCardHolder</class>
<class>com.nxsystems.simulator.idt.domain.IDTProgram</class>
<class>com.nxsystems.simulator.idt.domain.IDTTransaction</class>
<exclude-unlisted-classes>true</exclude-unlisted-classes>
<properties>
<!-- -->
<property name="hibernate.dialect" value="org.hibernate.dialect.HSQLDialect"/>
<property name="hibernate.default_schema" value="simulators"/>
<!-- -->
<property name="hibernate.show_sql" value="true"/>
<property name="hibernate.format_sql" value="true"/>
<!-- -->
<property name="hibernate.cache.use_second_level_cache" value="true"/>
<property name="hibernate.cache.use_query_cache" value="false"/>
<property name="hibernate.cache.region.factory_class" value="net.sf.ehcache.hibernate.EhCacheRegionFactory"/>
<!-- -->
<!--<property name="hibernate.hbm2ddl.auto" value="create"/>-->
<property name="hibernate.hbm2ddl.auto" value="create-drop"/>
<property name="hibernate.connection.pool_size" value="10"/>
</properties>
</persistence-unit>
</persistence>
And here are my init file script
DROP SCHEMA IF EXISTS simulators CASCADE;
CREATE SCHEMA SIMULATORS AUTHORIZATION SA -- not work yet
CREATE TABLE SIMULATORS.dirtyhack();
Maybe it would be better to create schema automatically, but I don't know how
UPDATE
I've founf how to launch db init script:
jdbc:initialize-database data-source="idtSimulatorDataSource">
Also I set
<property name="hibernate.hbm2ddl.auto" value="create"/>
But now I've got following error:
11:15:57,337 WARN (main) [JDBCExceptionReporter] SQL Error: -5501, SQLState: 42501
11:15:57,370 ERROR (main) [JDBCExceptionReporter] user lacks privilege or object not found: IDT_CARD
http://publib.boulder.ibm.com/iseries/v5r1/ic2924/index.htm?info/rzala/rzalastc.html
SQLSTATE Value: 42501
Meaning: The authorization ID does not have the privilege to perform the specified operation on the identified object.
What are privileges ?????
UPDATE:
Now I'm really confused.
I look into log. And see following:
/C2/SET SCHEMA PUBLIC DROP SCHEMA IF EXISTS simulators CASCADE
CREATE SCHEMA SIMULATORS AUTHORIZATION SA CREATE TABLE
SIMULATORS.dirtyhack() DISCONNECT /C3/SET SCHEMA PUBLIC DISCONNECT
/C4/SET SCHEMA PUBLIC DROP SCHEMA IF EXISTS simulators CASCADE
CREATE SCHEMA SIMULATORS AUTHORIZATION SA DISCONNECT /C5/SET SCHEMA
PUBLIC DISCONNECT /C6/SET SCHEMA PUBLIC
But I recently changed my init script:
DROP SCHEMA IF EXISTS simulators CASCADE;
CREATE SCHEMA SIMULATORS AUTHORIZATION sa -- not work yet
CREATE TABLE SIMULATORS.dirtyhack1();
Where is my dirtyhack1 () table???
I deleted all mentions about previous launches
UPDATE
I've found the reason.
This block is executed after Hibernate create its DDL manipulations:
<jdbc:embedded-database id="idtSimulatorDataSource" type="HSQL">
<jdbc:script location="classpath:hsql/idt_schema.sql"/>
</jdbc:embedded-database>
And my code contains such lines:
public static Server server = new Server();
#AfterClass
public static void tearDown1() throws Exception {
server.shutdown();
}
#BeforeClass
public static void setUp() throws Exception {
HsqlProperties p = new HsqlProperties();
p.setProperty("server.database.0", "file:./idt_simulatordb");
p.setProperty("server.dbname.0", "idt_simulatordb");
p.setProperty("server.port", "9001");
server.setProperties(p);
server.start();
Connection conn = DriverManager.getConnection("jdbc:hsqldb:hsql://localhost:9001/idt_simulatordb", "sa", "");
conn.setAutoCommit(true);
Statement st = conn.createStatement(); // statements
String expression1 = "DROP SCHEMA IF EXISTS simulators CASCADE;\n";
(1)String expression2 = "CREATE SCHEMA SIMULATORS AUTHORIZATION SA;";
String expression3 = "CREATE TABLE SIMULATORS.dirtyhack1();";
st.executeUpdate(expression1); // run the query
st.executeUpdate(expression2); // run the query
st.executeUpdate(expression3); // run the query
st.close();
conn.close();
}
That's why I didn't find my dirtyhack1 table, and that's why I got error. Init-database just removed my simulators schema
So there is a question. How to launch init script before Hibernate? Cause if I wouldn't write line (1) then I get error about wrong schema export
You could use the Spring EmbeddedDatabaseBuilder to create the database. Something like this:
new EmbeddedDatabaseBuilder().addScript("classpath:hsql/idt_schema.sql").build();
Then you don't need Hibernate to create the schema for you because it will already exist.