I have three entities, in which i try to save only 1 entity right now. All the three entities are shown below :-
1. Student Entity
<class name="com.school.Student" table="TABLE_STUDENT">
<id name="id" type="long">
<column name="ST_ID" />
<generator class="native" />
</id>
<property name="name" type="string" column="ST_NAME"/>
<many-to-one name="studentSection" class="com.school.Section" fetch="select">
<column name="SECTION_ID" not-null="true" />
</many-to-one>
<many-to-one name="studentSportsTeam" class="com.school.SportsTeam" fetch="select">
<column name="SPORTS_TEAM" not-null="true" />
</many-to-one>
</class>
2. Section Entity
<class name="com.school.Section" table="TABLE_SECTION">
<id name="sectionId" type="string">
<column name="SECTION_ID" />
<generator class="assigned" />
</id>
<property name="floor" type="string" column="SEC_FLOOR"/>
<property name="capcacity" type="int" column="SEC_CAPACITY"/>
<set name="studentDetails" inverse="true" lazy="true" table="TABLE_STUDENT" fetch="select">
<key>
<column name="SECTION_ID" not-null="true" />
</key>
<one-to-many class="com.school.Student" />
</set>
</class>
3. SprotsTeam Entity :-
<class name="com.school.SportsTeam" table="TABLE_SPORTS">
<id name="sportsTeamId" type="string">
<column name="SPORTS_TEAM" />
<generator class="assigned" />
</id>
<property name="noOfPlayers" type="int" column="SPORTS_PLAYER_NUM"/>
<property name="captainName" type="string" column="SPORTS_CAPTAIN_NAME"/>
<set name="playerDetails" inverse="true" lazy="true" table="TABLE_STUDENT" fetch="select">
<key>
<column name="SPORTS_TEAM" not-null="true" />
</key>
<one-to-many class="com.school.Student" />
</set>
</class>
Now if i try to save Student Entity with proper Section and SportsTeam details, it takes a lot of time to persist it into the database. Currently i am running it for around 10000 students and this process (only persisting) takes around 15 mins. I added some loggers to calculate the complete time.
Now i need to reduce this time, as we will shorty move from 10,000 to 1 million records, and as calculated it takes very long time.. I need to reduce the time , how can i do that??
As Required, also the schema is as below :-
TABLE STUDENT :
ST_ID NUMBER,
ST_NAME VARCHAR(40),
SECTION_ID VARCHAR(10),
SPORTS_TEAM VARCHAR(10)
TABLE_SECTION :
SECTION_ID VARCHAR(10),
SEC_FLOOR VARCHAR(2),
SEC_CAPACITY NUMBER
TABLE_SPORTS :
SPORTS_TEAM VARCHAR(10),
SPORTS_PLAYER_NUM NUMBER,
SPORTS_CAPTAIN_NAME VARCHAR(40)
Please help
Consider batch inserts.
Moreover, if this batch inserts is just for once i.e. as a part of migration process, then IMO, you can consider dropping all primary keys -- and any other indices, on the tables, and then insert the records. After that re-create all those. Hopefully, you would notice a significant improvement.
I think you are trying to import the Student Entity with Section and SportsTeam. At this time if you set the values from the input data to the elements of the Entities and call save on Student entity then it would result into creation of a Student, Section and SportsTeam records in the data base. Which means if you save 10000 student entity then you are effectvely creating 10000 SportsTeam record and 10000 Section record.
Instead I would suggest you to follow these steps:
1. Read a line from the input data
2. query db (through hql) for the already exiting SportsTeam and Section
3. If no records found for SportsTeam and Section then create them
3. Create a new student record and set the SportsTeam and Section from step2.
4. Save the student record.
Also I would suggest you to optimize the creation and save of Students objects in a batch of say 1000 in a single transaction. Closing the hibernate transaction after a batch and releasing the objects would help increase the utilization of DB/Network as well as memory.
I would specify a length for string properties; otherwise they may be implemented as clobs
Related
Trying to insert a new Entity using hibernate and it is throwing me this exception:
a different object with the same identifier value was already associated with the session
I understand that this error is coming because hibernate finds a similar object in the memory. But I am creating a new object every time before inserting. Does it have anything to do with sequence?
hbm
<class name="MyObject" table="My_Object">
<id column="object_id" name="id" type="long">
<generator class="sequence">
<param name="sequence">OBJ_SEQ</param>
</generator>
</id>
<property name="column1" column="column_1" type="string" not-null="true"/>
<property name="column2" column="column_2" type="string" not-null="true"/>
<property name="column3" column="column_3" not-null="true" type="string"/>
</class>
The problem is when you are trying to persist an entity, a entity with the same id is already present in the database. Reason for this can be, you are manually assigning a value to the id, or you have reset the OBJ_SEQ in database.
I need to process big Excel files. My problem is heap space especially with XLS format. Even if I have one Apsose LighCells API to parse it sequentially I need to retrieve the file from the database in chuncks.
So far I have the file in the database in chunks of 40kb. I have an Import table (storing general information related to the import, for instance start and end time, data type etc.) and an ImportData table (containing the chunks of data as blobs). I have a one to many relation between Import and ImportData:
<hibernate-mapping> <class name="com.company.import.pojos.Import
table="IMPORT_TABLE">
<id name="id" type="integer">
<column name="ID" />
<generator class="some.id.generator.IdGenerator"></generator>
</id>
<property name="startTime" type="timestamp">
<column name="START" />
</property>
<property name="endTime" type="timestamp">
<column name="END" />
</property>
<property lazy="false" name="datatzpe" type="com.company.import.enums.ImportDataType">
<column name="DATATYPE" />
</property>
<bag name="importDataList" table="IMPORT_DATA" lazy="true" cascade="all" inverse="false">
<key column="IMPORT_TABLE_ID"/>
<one-to-many class="com.company.import.pojos.ImportData"/>
</bag>
</class> </hibernate-mapping>
<hibernate-mapping> <class
name="com.company.import.pojos.ImportData" table="IMPORT_DATA">
<id name="id" type="integer">
<column name="ID" />
<generator class="some.id.generator.IdGenerator"></generator>
</id>
<property name="importTableID" type="integer">
<column name="IMPORT_TABLE_ID" />
</property>
<property name="data" type="binary">
<column name="DATA" />
</property>
<property name="order" type="integer">
<column name="ORDER" />
</property> </class> </hibernate-mapping>
I use Hibernate (4.2.3.Final), but even lazy loading in not an option because then I will have the whole list of ImportData objects in memory which is why I get an OutOfMemoryError and I want to avoid that.
Providing more memory is not an option (the application runs ona container that runs other applications as well, and several users might process files at the same time so the heaps space problem will eventually happen), therefore I am looking for a way to always read the next chunk of data and provide it to the stream that is given to the Aspose API.
I create my workbook in Aspose simply like this:
InputStream inStream = ....;// get the input stream somehow
Workbook workbook = new Workbook(inStream);
You may need to implement your own InputStream, which handles multiple IS as one, handling that when you reach the end of one IS you close it and pick the next one to fill the rest of the read buffer.
I do not know any implementation for this behaviour.
How big is your file ?
I wanna set lazy to "false" but only to one method at in runtime.
Can I do?
this.getSession().createSQLQuery("select * from customers....")....
Attention: im using createSQLQuery not createCriteria.
CustomerMapping.xml here:
<hibernate-mapping>
<class name="com.example.entities.customers.Customer"
table="CUSTOMERS">
<id name="id" type="long">
<column name="ID" />
<generator class="sequence">
<param name="sequence">seq_customers</param>
</generator>
</id>
<property name="name" type="String">
<column name="NAME_C" />
</property>
<many-to-one name="address"
class="com.example.entities.Address" fetch="select"
cascade="all">
<column name="ADDRESS_ID" />
</many-to-one>
</class>
</hibernate-mapping>
I wanna set lazy to false for Address.
I have to do this because this method return a list of customers (with address) and when I iterate this list and print its very slow cause the lazy is setted true (by default).
Is there a reason you are using SQL instead of HQL? I would stay away from SQL statements when using hibernate when possible.
I would implement it like this in HQL:
from Customer c
join fetch c.address
the join fetch makes Customers address no longer lazy.
---- Introductionary information and problem domain ----
Basicly I have 3 tables in a database: 'User', 'Item', 'ItemsPerUser'.
Table User:
username (PK);
password;
email
Table Item
name (PK)
Table ItemsPerUser
username (PK) (and FK);
item_name (PK) (and FK)
When I don't use cascading, I get an error: "Cannot add or update a child row: a foreign key constraint fails".
The mapping files are correct. I need some sort of cascading. It works when I add cascading in the set property of the many-to-many relationship to add non-existent data to User & Item, but it is overwriting data in ItemsPerUser. Whenever I save an object which contains one ore more items which was already entered in ItemsPerUser, it overwrites the row, even when the other part of the PK is not the same user. So basicly the previous user with that 'item' is overwritten by the new user with the same item.
It should always add a new row in the table ItemsPerUser if it is a new user, even with one or more item(s) whom is already entered by another User object.
---- Visual styled example ----
Assume I start with an empty database and I insert a new user Roger, who has two items: coffee and water. This is an example what happens (Hibernate handles this correct):
User ItemsPerUser Item
Roger Roger-coffee coffee
Roger-water water
Now when I insert a new user "Alfonzo" whom has the items coffee and soda, this happens:
User ItemsPerUser Item
Roger Alfonzo-coffee coffee
Alfonzo Roger-water water
Alfonzo-soda soda
---- Code example(s) ----
//Mapping for databag 'User' - !! NOTE: I have deleted the cascade rule in the XML
<hibernate-mapping>
<class name="databag.User" table="User" catalog="androiddb">
<id name="username" type="string">
<column name="username" length="45" />
<generator class="assigned" />
</id>
<property name="password" type="string">
<column name="password" length="45" not-null="true" />
</property>
<property name="email" type="string">
<column name="email" length="45" not-null="true" unique="true" />
</property>
<set name="items" inverse="false" table="itemsperuser">
<key>
<column name="username" length="45" not-null="true" />
</key>
<many-to-many entity-name="databag.Items">
<column name="item_name" length="45" not-null="true"/>
</many-to-many>
</set>
</class>
</hibernate-mapping>
//Mapping for Item
<hibernate-mapping>
<class name="databag.Item" table="item" catalog="androiddb">
<id name="name" type="string">
<column name="name" length="45" />
<generator class="assigned" />
</id>
<set name="users" inverse="false" table="itemsperuser">
<key>
<column name="item_naam" length="45" not-null="true" />
</key>
<many-to-many entity-name="databag.User">
<column name="username" length="45" not-null="true" />
</many-to-many>
</set>
</class>
</hibernate-mapping>
//Saving an object
Session session = HibernateUtil.getSessionFactory().openSession();
Transaction trans = session.beginTransaction();
session.save((User)o);
trans.commit();
session.close();
Note: '(User)o' contains none, one or more items.
You can't set inverse="false" on both sides. Hibernate cannot persists both sets (users in Item and items in User) in the same table, unless one of them is called 'inverse', and can be savely ignored for writing. It will be filled up only when reading.
I'd set inverse="true" on the Item side.
Then you'd have to save items first, then adding them to a user, then saving the user.
I am using Hibernate. My database is as follows
A Category has many attributes
class category
contains
private Set <Attribute> AllAttributes= new HashSet <Attribute>();
class attribute
How do I retrieve all categories together with their attributes because I am trying 'from category' but it is not working
Category Mapping File:
<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC "-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
<!-- Generated Dec 16, 2010 8:37:02 AM by Hibernate Tools 3.4.0.Beta1 -->
<hibernate-mapping>
<class name="com.BiddingSystem.Models.Category" table="CATEGORY">
<id name="CategoryId" type="long">
<column name="CATEGORYID" />
<generator class="native" />
</id>
<property name="CategoryName" type="java.lang.String">
<column name="CATEGORYNAME" />
</property>
<many-to-one name="ParentCategory" class="com.BiddingSystem.Models.Category">
<column name="PARENT_CATEGORY_ID" />
</many-to-one>
<set name="SubCategory" lazy="false" cascade="all-delete-orphan" inverse="true">
<key>
<column name="PARENT_CATEGORY_ID" />
</key>
<one-to-many class="com.BiddingSystem.Models.Category" />
</set>
<set name="AllAttributes" table="ATTRIBUTE" inverse="false" lazy="true" cascade="all">
<key>
<column name="CATEGORYID" />
</key>
<one-to-many class="com.BiddingSystem.Models.Attribute" />
</set>
</class>
</hibernate-mapping>
Attribute Mapping File:
<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC "-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
<!-- Generated Dec 16, 2010 5:25:09 AM by Hibernate Tools 3.4.0.Beta1 -->
<hibernate-mapping>
<class name="com.BiddingSystem.Models.Attribute" table="ATTRIBUTE">
<id name="AttributeId" type="long">
<column name="ATTRIBUTEID" />
<generator class="native" />
</id>
<property name="AttributeName" type="java.lang.String">
<column name="ATTRIBUTENAME" />
</property>
<set name="Options" table="ATTRIBUTEOPTION" inverse="false" cascade="all">
<key>
<column name="ATTRIBUTEID" />
</key>
<one-to-many class="com.BiddingSystem.Models.AttributeOption" />
</set>
</class>
</hibernate-mapping>
You have mapped the association with lazy="true". This tells hibernate that by default, a category's attributes should only be loaded from the database when they are actually accessed, in contrast to lazy="false" which would instruct hibernate to load the attributes whenever it loads a category. However, the directive in the mapping file affects all queries.
In case you want it only for a particular query, check out
http://docs.jboss.org/hibernate/core/3.5/reference/en/html/queryhql.html#queryhql-joins :
A "fetch" join allows associations or
collections of values to be
initialized along with their parent
objects using a single select. This is
particularly useful in the case of a
collection. It effectively overrides
the outer join and lazy declarations
of the mapping file for associations
and collections. See Section 20.1,
“Fetching strategies” for more
information.
from Cat as cat
inner join fetch cat.mate
left join fetch cat.kittens
A fetch join does not usually need to
assign an alias, because the
associated objects should not be used
in the where clause (or any other
clause). The associated objects are
also not returned directly in the
query results. Instead, they may be
accessed via the parent object. The
only reason you might need an alias is
if you are recursively join fetching a
further collection:
from Cat as cat
inner join fetch cat.mate
left join fetch cat.kittens child
left join fetch child.kittens
The fetch construct cannot be used in
queries called using iterate() (though
scroll() can be used). Fetch should be
used together with setMaxResults() or
setFirstResult(), as these operations
are based on the result rows which
usually contain duplicates for eager
collection fetching, hence, the number
of rows is not what you would expect.
Fetch should also not be used together
with impromptu with condition. It is
possible to create a cartesian product
by join fetching more than one
collection in a query, so take care in
this case. Join fetching multiple
collection roles can produce
unexpected results for bag mappings,
so user discretion is advised when
formulating queries in this case.
Finally, note that full join fetch and
right join fetch are not meaningful.