Now I'm trying to enhance performance of my web application, I use spring JPA 2.3.0- Hibernate 5.4.15.Final, Postgres 12 and manage transaction by #Transaction. The web app is deployed on aws beanstalk, run multiple instances at the same time, but the database instance is not scalable. And I use bigSerial type for ID of tables.
For instance, I have a STUDENTS table, ID is bigSerial and some other columns.
I got a problems when using
#GeneratedValue(strategy = GenerationType.IDENTITY)
,
Hibernate couldn't batch insert when saving a list of entities.
And I try to use
#GeneratedValue(strategy = GenerationType.AUTO, generator = "students_id_seq")
#SequenceGenerator(name = "students_id_seq", sequenceName = "students_id_seq")
hibernate.id.new_generator_mappings=false
hibernate.jdbc.batch_size=10
hibernate.order_inserts=true
hibernate.order_updates=true
hibernate.batch_versioned_data=true
It seem Hibernate could batch insert, but the problem is Hibernate execute select nextval ('students_id_seq') multiple times. If an entity list has 30 items, Hibernate execute select nextval 30 times, and 3 times for batch insert query.
Some statistics:
If using GenerationType.IDENTITY
save(entity):
insert into ... : execute once
saveAll(n entities)
insert into ... : execute n times
If using GenerationType.SEQUENCE/ GenerationType.AUTO
save(entity):
select nextval ('students_id_seq'): execute once
insert into ... : execute once
saveAll(n entities):
select nextval ('students_id_seq'): execute n times
insert into ... : execute n/batch_size times
In conclusion, If using GenerationType.AUTO or GenerationType.SEQUENCE with allocationSize = 1:
when insert one entity, application increases 100% times to execute
queries ( from one insert query only increase to 2 queries: select
nextval, and insert query )
when batch insert, application increase more than 10% if batch_size = 10
My question is, is there anyway to batch insert but not execute to many select nextval query ? Something likes GenerationType.IDENTITY, not execute select nextval, just batch insert and IDs will be handled by sequence in the database.
When I test with GenerationType.SEQUENCE and allocationSize=1 (GenerationType.AUTO), the application executes too much select nextval query, I think It is even worse than the IDENTITY strategy.
And for some reasons, I don't want to use allocationSize, it may lead to duplicate primary key error when run insert query manual or when migrate data or some other cases.
After some research, I found a way to get a value list of a sequence:
select nextval ('students_id_seq') from generate_series(1,10);
We can replace 10 by entityList.size() or number of entities doesn't have ID in the entityList when batch insert, just get enough to use, don't create too much gap between IDs, but I'm not sure whether or not Hibernate supported, if yes, please share me the documentation to reference.
Thank you
https://discourse.hibernate.org/t/batch-insert-execute-too-much-select-nextval-sequence/4232
What you are looking for is the HiLo algorithm for id generation.
For every id generated from a sequence it generates a number of ids on the client without accessing the database.
You configure it on your entity as this:
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "hilo_sequence_generator")
#GenericGenerator(
name = "hilo_sequence_generator",
strategy = "org.hibernate.id.enhanced.SequenceStyleGenerator",
parameters = {
#Parameter(name = "sequence_name", value = "hilo_seqeunce"),
#Parameter(name = "initial_value", value = "1"),
#Parameter(name = "increment_size", value = "3"),
#Parameter(name = "optimizer", value = "hilo")
})
#Id
private Long id;
I would say that I have some experience of that point. I was doing insertion for more than 128,000 records.
And my target was to enhance the time-consuming to do that. I would try to summarize the case as below :
The code did not use any persist() or save() methods. These records were saved when the #Transactionl method exits
I was using hibernate batch inserts below are the settings in the config map
spring.jpa.properties.hibernate.jdbc.batch_size: "40"
spring.jpa.properties.hibernate.order_inserts: "true"
spring.jpa.properties.hibernate.order_updates: "true"
spring.main.allow-bean-definition-overriding: "true"
I modified the allocation size in my entity Id configuration as below:
#Id
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator ="mappingentry_sql_generator")
#SequenceGenerator(name = "mappingentry_sql_generator", sequenceName ="mappingentry_id_seq", allocationSize = 40)
private Long id;
Notes: I set the "allocationSize" in the sequence generator to be equal to the "batch_size" value in the setting
Also, I altered my sequence "mappingentry_id_seq" to increment by 40 as well
After doing these changes the time was reduced from 55 seconds to 20 seconds which was a great impact
The only thing that I did not understand is that when I checked the value of the id column that was generated by the sequence I did not find any value gap. Every ID value exceeded the previous by 1, not 40. That is what I am currently trying to understand
Related
We are using #sequencegenerator annotations to generate sequence in mssql database but when we execute sequence is generated as table instead of sequence, we have used 2012Dialect but still we face same issue and application throws exception invalid object name - sequence name-. Please help with solution
Have you used GeneratedValue annotation to specify that the id generation value witll be from a database sequence?
example:
#Id
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "seq_name")
#SequenceGenerator(name = "seq_name", sequenceName = "seq_name_in_database", allocationSize = 1)
#Column(name = "id")
private Long id;
It seems that Hibernate (v 5.6.3) is using its own tables to manage sequences in sql server, you must create the table with the same name as your sequence with one column named next_val then insert a line with the init value.
Hibernate use the query to select :
select next_val as id_val from sequence_name with (updlock,rowlock)
then increment the next_val :
update sequence_utilisateur set next_val= ? where next_val=?
Also don't define the column for wich you use the generated value as IDENTITY.
We have an application which is working for years and we are using same oracle database too. But migrated our database from one host to another host.
DB: ORACLE
Now all of the sudden we are getting following exception,
“org.springframework.dao.DataIntegrityViolationException: ORA-00001: unique constraint (YYY.XXX_LOG_PK) violated;
SQL [n/a]; constraint [YYY.XXX_LOG_PK]; nested exception is org.hibernate.exception.ConstraintViolationException: ORA-00001: unique constraint (YYY.XXX_LOG_PK) violated”
code:
#SequenceGenerator(name = "TT_SEQUENCE_GENERATOR", sequenceName = "YYY.XXX_LOG_SEQ")
#Id
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "TT_SEQUENCE_GENERATOR")
#Column(name = "ID")
public Long getId() {
return this.id;
}
Sequence in DB:
CREATED 07-NOV-17
LAST_DDL_TIME 07-NOV-17
SEQUENCE_OWNER TT
SEQUENCE_NAME YYY.XXX_LOG_SEQ
MIN_VALUE 1
MAX_VALUE 999999999999999999999999999
INCREMENT_BY 1
CYCLE_FLAG N
ORDER_FLAG N
CACHE_SIZE 0
LAST_NUMBER 75305
Problem:
When we are trying to insert some record through JPA code we are getting the above exception but when I try to insert some record into the DB using sequence.nextval, it is not giving any exception.
Is there anyway I can debug to find out what would be the exception ? I also tried show_sql - I couldn't able to find the solution with this too, as this doesn't print the next sequence number in the console
Please point me in right direction, if you know the solution.
The most common scenario in which a self-augmented sequence encounters a unique constraint conflict is when the data is migrated, causing the maximun value of the data to exceed the sequence value.
Firstly query the sequanence current values:
SELECT seqname.CURRVAL FROM dual
And then modify the sequence value to make sure the sequence's nextval exceeds the current maximum value of the data.
ALTER SEQUENCE seqname INCREMENT BY XXXXXX;
SELECT seqname.NEXTVAL FROM dual;
I found the solution for the problem which we faced.
The problem is from the database end. We found that by reviewing the migration documents and by checking the files.
#Edwin: Thanks for your help, your query also helped me in finding where is the problem residing.
While doing migration the sequences haven't copied from old server to new server. When we copied to new server, everything worked fine.
Thanks everyone.
I'm using Postgres database in production and H2 for tests.
I want to create a new sequence for an existing table - so in Liquibase I wrote this:
<changeSet id="Add sequence for BOOKS" author="library">
<createSequence sequenceName="BOOKS_SEQ" incrementBy="500" startValue="1"/>
</changeSet>
My Entity looks like this:
#Entity
#Table(name = "BOOKS")
#SequenceGenerator(name = "BOOKS_SEQ", allocationSize = 500)
public class Book {
#Id
#GeneratedValue(generator = "BOOKS_SEQ", strategy = GenerationType.TABLE)
private Long id;
#ManyToOne
#JoinColumn(name = "AUTHOR_ID")
private Author author;
...
}
Now, since I already have entities in this table (with Ids from the last sequence I used), I need to set the current value of this sequence accordingly.
For that matter I wrote:
<changeSet id="Alter sequence for BOOKS" author="library">
<sql dbms="postgresql">select setval('BOOKS_SEQ', (select nextval('OLD_SEQUENCE_UID')))</sql>
</changeSet>
<changeSet id="Add default value to BOOKS.ID" author="library">
<addDefaultValue tableName="BOOKS" columnName="ID" defaultValueSequenceNext="BOOKS_SEQ"/>
</changeSet>
In Postgres (production), it seems to work just fine - but in H2 I get an error message "The sequence named [BOOKS_SEQ] is setup incorrectly. Its increment does not match its pre-allocation size".
According to this - I need to set the start_value (or current value) to something greater than 500 - but I can't figure out how this can be done in H2.
So my question is: How can I set the current value of a sequence in H2?
Have you tried with this?
alter sequence <your_sequence_name> restart with <next_value>
For example:
alter sequence BOOKS_SEQ restart with 500
When I run the above command, I have BOOKS_SEQ set its current value to 499 (so the next value will be 500).
I have a standard JPA #SequenceGenerator annotated entity:
#Id
#Column(name = "ID", unique = true, nullable = false)
#GeneratedValue(generator = "mySeq", strategy = GenerationType.SEQUENCE)
#SequenceGenerator(name = "mySeq", sequenceName = "MY_SEQ", allocationSize = 10)
private long id;
with the named Oracle Sequence defined as:
CREATE SEQUENCE MY_SEQ
START WITH 55554444
INCREMENT BY 1
NOORDER NOCYCLE;
When I persist through a standard Java, Spring Data Service, the first persisted, generated ID value is:
555544440
In other words, it's * 10 what I had defined as a starting value.
Is this expected behaviour?
When I query:
select last_number from dba_sequences
though, the returned value is still of the 55554444 range.
Running an equivalent insert directly on the DB
INSERT INTO MY_TABLE (ID) VALUES (MY_SEQ.nextVal)
the ID value is generated and persisted as I would expect; i.e 55554444, 55554445, 55554446, etc. (and also correlates the last_number in dba_sequences)
Whats going on! How and why is the JPA persistence * 10 my sequence IDs!?
Stumped, any help appreciated!
thanks,
Damien
Hibernate, #SequenceGenerator and allocationSize answered this.
That is, if the JPA defined allocationSize differs from the oracle sequence increment size, bad things happen.
For me:
JPA allocationSize = 10
DB sequence increment = 1
Java persistence was then multiplying the oracle sequence * 10
I have three entity:
Users(id, username, password...); BoughtItems(id, userid, itemcat, itemid); ItemCustomizations(id(id of item), slot(id of color), color)
When i fetch like this in BoughtItems.java:
#OneToMany(fetch = FetchType.EAGER, mappedBy = "id")
#OrderBy("slot")
private List<ItemCustomization> customColors = new ArrayList<>();
Hibernate asks database for each item:
Hibernate: select customcolo0_.itemid as itemid1_2_0_, customcolo0_.itemid
as itemid1_3_0_, customcolo0_.slot as slot2_3_0_, customcolo0_.itemid as
itemid1_3_1_, customcolo0_.slot as slot2_3_1_, customcolo0_.color as
color3_3_1_ from shop_customizations customcolo0_ where
customcolo0_.itemid=? order by customcolo0_.slot
My question
How to proper optimize my mappings to reduce mysql server performance hit?
Can I somehow optimize query/item to 1 query?
I believe you are looking for batch size feature of hibernate. See example here.
Batch size defines how many collections should be loaded. For example if you have 10 BoughtItems in session with batch size =5 , only two sql queries will be fired to fetch customColors collections for all possible BoughtItems
See Possible fetching strategies in hibernate