Develop Reference

Spring JPA - Hibernate: Batch insert execute too much select nextval (‘sequence’)

Now I'm trying to enhance performance of my web application, I use spring JPA 2.3.0- Hibernate 5.4.15.Final, Postgres 12 and manage transaction by #Transaction. The web app is deployed on aws beanstalk, run multiple instances at the same time, but the database instance is not scalable. And I use bigSerial type for ID of tables.
For instance, I have a STUDENTS table, ID is bigSerial and some other columns.
I got a problems when using
#GeneratedValue(strategy = GenerationType.IDENTITY)
,
Hibernate couldn't batch insert when saving a list of entities.
And I try to use
#GeneratedValue(strategy = GenerationType.AUTO, generator = "students_id_seq")
#SequenceGenerator(name = "students_id_seq", sequenceName = "students_id_seq")
hibernate.id.new_generator_mappings=false
hibernate.jdbc.batch_size=10
hibernate.order_inserts=true
hibernate.order_updates=true
hibernate.batch_versioned_data=true
It seem Hibernate could batch insert, but the problem is Hibernate execute select nextval ('students_id_seq') multiple times. If an entity list has 30 items, Hibernate execute select nextval 30 times, and 3 times for batch insert query.
Some statistics:
If using GenerationType.IDENTITY
save(entity):
insert into ... : execute once
saveAll(n entities)
insert into ... : execute n times
If using GenerationType.SEQUENCE/ GenerationType.AUTO
save(entity):
select nextval ('students_id_seq'): execute once
insert into ... : execute once
saveAll(n entities):
select nextval ('students_id_seq'): execute n times
insert into ... : execute n/batch_size times
In conclusion, If using GenerationType.AUTO or GenerationType.SEQUENCE with allocationSize = 1:
when insert one entity, application increases 100% times to execute
queries ( from one insert query only increase to 2 queries: select
nextval, and insert query )
when batch insert, application increase more than 10% if batch_size = 10
My question is, is there anyway to batch insert but not execute to many select nextval query ? Something likes GenerationType.IDENTITY, not execute select nextval, just batch insert and IDs will be handled by sequence in the database.
When I test with GenerationType.SEQUENCE and allocationSize=1 (GenerationType.AUTO), the application executes too much select nextval query, I think It is even worse than the IDENTITY strategy.
And for some reasons, I don't want to use allocationSize, it may lead to duplicate primary key error when run insert query manual or when migrate data or some other cases.
After some research, I found a way to get a value list of a sequence:
select nextval ('students_id_seq') from generate_series(1,10);
We can replace 10 by entityList.size() or number of entities doesn't have ID in the entityList when batch insert, just get enough to use, don't create too much gap between IDs, but I'm not sure whether or not Hibernate supported, if yes, please share me the documentation to reference.
Thank you
https://discourse.hibernate.org/t/batch-insert-execute-too-much-select-nextval-sequence/4232

What you are looking for is the HiLo algorithm for id generation.
For every id generated from a sequence it generates a number of ids on the client without accessing the database.
You configure it on your entity as this:
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "hilo_sequence_generator")
#GenericGenerator(
name = "hilo_sequence_generator",
strategy = "org.hibernate.id.enhanced.SequenceStyleGenerator",
parameters = {
#Parameter(name = "sequence_name", value = "hilo_seqeunce"),
#Parameter(name = "initial_value", value = "1"),
#Parameter(name = "increment_size", value = "3"),
#Parameter(name = "optimizer", value = "hilo")
})
#Id
private Long id;

I would say that I have some experience of that point. I was doing insertion for more than 128,000 records.
And my target was to enhance the time-consuming to do that. I would try to summarize the case as below :
The code did not use any persist() or save() methods. These records were saved when the #Transactionl method exits
I was using hibernate batch inserts below are the settings in the config map
spring.jpa.properties.hibernate.jdbc.batch_size: "40"
spring.jpa.properties.hibernate.order_inserts: "true"
spring.jpa.properties.hibernate.order_updates: "true"
spring.main.allow-bean-definition-overriding: "true"
I modified the allocation size in my entity Id configuration as below:
#Id
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator ="mappingentry_sql_generator")
#SequenceGenerator(name = "mappingentry_sql_generator", sequenceName ="mappingentry_id_seq", allocationSize = 40)
private Long id;
Notes: I set the "allocationSize" in the sequence generator to be equal to the "batch_size" value in the setting
Also, I altered my sequence "mappingentry_id_seq" to increment by 40 as well
After doing these changes the time was reduced from 55 seconds to 20 seconds which was a great impact
The only thing that I did not understand is that when I checked the value of the id column that was generated by the sequence I did not find any value gap. Every ID value exceeded the previous by 1, not 40. That is what I am currently trying to understand

How / where does Oracle set / store #SequenceGenerator values

I have a standard JPA #SequenceGenerator annotated entity:
#Id
#Column(name = "ID", unique = true, nullable = false)
#GeneratedValue(generator = "mySeq", strategy = GenerationType.SEQUENCE)
#SequenceGenerator(name = "mySeq", sequenceName = "MY_SEQ", allocationSize = 10)
private long id;
with the named Oracle Sequence defined as:
CREATE SEQUENCE MY_SEQ
START WITH 55554444
INCREMENT BY 1
NOORDER NOCYCLE;
When I persist through a standard Java, Spring Data Service, the first persisted, generated ID value is:
555544440
In other words, it's * 10 what I had defined as a starting value.
Is this expected behaviour?
When I query:
select last_number from dba_sequences
though, the returned value is still of the 55554444 range.
Running an equivalent insert directly on the DB
INSERT INTO MY_TABLE (ID) VALUES (MY_SEQ.nextVal)
the ID value is generated and persisted as I would expect; i.e 55554444, 55554445, 55554446, etc. (and also correlates the last_number in dba_sequences)
Whats going on! How and why is the JPA persistence * 10 my sequence IDs!?
Stumped, any help appreciated!
thanks,
Damien

Hibernate, #SequenceGenerator and allocationSize answered this.
That is, if the JPA defined allocationSize differs from the oracle sequence increment size, bad things happen.
For me:
JPA allocationSize = 10
DB sequence increment = 1
Java persistence was then multiplying the oracle sequence * 10

Hipernate won't update object, resets primary key to 0

I'm using JavaServer Pages, Hibernate and MySql.
I have an update page (form) for an object from the database, but when I try and update it I receive the following massage:
Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1
I've put some System.out.println functions to try and check where the error is and I've come to the conclusion that the primary key is reset to 0 when going in the ManagedBean, but only primary key value is reset, all other field values are the same.
Perhaps if would be useful to say that the primary key is mapped (with hibernate) as:
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
Does anyone know what could be the problem?

INSERT with DEFAULT id doesn't work in PostgreSQL

I tried running this statement in Postgres:
insert into field (id, name) values (DEFAULT, 'Me')
and I got this error:
ERROR: null value in column "id" violates not-null constraint
I ended up having to manually set the id. The problem with that is when my app inserts a record I get a duplicate key error. I am building a java app using Play framework and ebean ORM. So the entire schema is generated automatically by ebean. In this case, what is the best practice for inserting a record manually into my db?
Edit:
Here is how I'm creating my Field class
#Entity
public class Field {
#id
public Long id;
public String name;
}
Edit:
I checked the field_seq sequence and it looks like this:
CREATE SEQUENCE public.field_seq INCREMENT BY 1 MINVALUE 1 MAXVALUE 9223372036854775807 START 1 CACHE 1;
Edit:
Here is the generated SQL in pgAdmin III:
CREATE TABLE field
(
id bigint NOT NULL,
created timestamp without time zone,
modified timestamp without time zone,
name character varying(255),
enabled boolean,
auto_set boolean,
section character varying(17),
input_type character varying(8),
user_id bigint,
CONSTRAINT pk_field PRIMARY KEY (id),
CONSTRAINT fk_field_user_3 FOREIGN KEY (user_id)
REFERENCES account (id) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT ck_field_input_type CHECK (input_type::text = ANY (ARRAY['TEXT'::character varying, 'TEXTAREA'::character varying]::text[])),
CONSTRAINT ck_field_section CHECK (section::text = ANY (ARRAY['MAIN_CONTACT_INFO'::character varying, 'PARTICIPANT_INFO'::character varying]::text[]))
);
CREATE INDEX ix_field_user_3
ON field
USING btree
(user_id);

There is no column default defined for field.id. Since the sequence public.field_seq seems to exist already (but is not attached to field.id) you can fix it with:
ALTER SEQUENCE field_seq OWNED BY field.id;
ALTER TABLE field
ALTER COLUMN id SET DEFAULT (nextval('field_seq'::regclass));
Make sure the sequence isn't in use for something else, though.
It would be much simpler to create your table like this to begin with:
CREATE TABLE field
(
id bigserial PRIMARY KEY,
...
);
Details on serial or bigserial in the manual.
Not sure how the the Play framework implements this.

This works.
insert into field (id, name) values (nextval('field_seq'), "Me");

Manually specify the value of a primary key in JPA #GeneratedValue column

I'm having an Entity which has a primary key / id field like the following:
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
This works well. I'm using EclipseLink to create the DDL-Schema, and the column is correctly created like so:
`id` bigint(20) NOT NULL AUTO_INCREMENT
However, I've got several entities for which I do want to specify the PK myself (it's a little application that transfers data from an old database to the new one we're building). If I specify the ID for the POJO (using setId(Long id)) and persist it, EclipseLink does not save it (i.e. the record is saved, but the id is auto generated by eclipseLink).
Is there a way to manually specify the value of a column which has a #GeneratedValue ?
Here some thoughts on the issue:
I tried to work around the problem by not using #GeneratedValue at all, but simply manually define the column to be AUTO_INCREMENTed. However this forces me to manually provide an IDs always, since EclipseLink validates the primary key (so it may not be null, zero, or a negative number). The exception message reads that I should specify eclipselink.id_validation, however this does not seem to make any difference (I annotated #PrimaryKey(validation = IdValidation.NONE) but still got the same message).
To clarify: I'm using EclipseLink (2.4.0) as persistence provider and I can't switch away from it (large portions of the project depend on eclipselink specific query hints, annotations, and classes).
EDIT (In Response to the answers):
Custom Sequencing: I tried to implement my own sequencing. I tried subclassing DefaultSequence, but EclipseLink will tell me Internal Exception: org.eclipse.persistence.platform.database.MySQLPlatform could not be found. But I've checked: The class is on the classpath.
So I subclassed another class, NativeSequence:
public class MyNativeSequence extends NativeSequence {
public MyNativeSequence() {
super();
}
public MyNativeSequence(final String name) {
super(name);
}
#Override
public boolean shouldAlwaysOverrideExistingValue() {
return false;
}
#Override
public boolean shouldAlwaysOverrideExistingValue(final String seqName) {
return false;
}
}
However, what I get is the following:
javax.persistence.RollbackException: Exception [EclipseLink-7197] (Eclipse Persistence Services - 2.4.0.v20120608-r11652): org.eclipse.persistence.exceptions.ValidationException
Exception Description: Null or zero primary key encountered in unit of work clone [de.dfv.datenbank.domain.Mitarbeiter[ id=null ]], primary key [null]. Set descriptors IdValidation or the "eclipselink.id-validation" property.
at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commitInternal(EntityTransactionImpl.java:102)
...
Caused by: Exception [EclipseLink-7197] (Eclipse Persistence Services - 2.4.0.v20120608-r11652): org.eclipse.persistence.exceptions.ValidationException
Exception Description: Null or zero primary key encountered in unit of work clone [de.dfv.datenbank.domain.Mitarbeiter[ id=null ]], primary key [null]. Set descriptors IdValidation or the "eclipselink.id-validation" property.
at org.eclipse.persistence.exceptions.ValidationException.nullPrimaryKeyInUnitOfWorkClone(ValidationException.java:1451)
...
(stack trace shortened for clarity). This is the same message which I got before. Shouldn't I subclass NativeSequence? If so, I don't know what to implement for the abstract methods in Sequence or StandardSequence.
It may also be worth noting, that simply subclassing (without overriding any methods) the class works as expected. However, returing false in shouldAlwaysOverrideExistingValue(...) will not generate a single value at all (I stepped through the program and getGeneratedValue() is not called once).
Also, when I insert like 8 entities of a certain kind within a transaction it resulted in 11 records in the database (what the hell?!).
EDIT (2012-09-01): I still do not have a Solution for the problem, Implementing my own sequence did not solve it. What I need is a way to be able to not set an Id explicitly (so it will be auto generated) and to be able to set an Id explicitly (so it will be used for the creation of the record in the database).
I tried to define the column as auto_increment myself and ommit #GeneratedValue, however Validation will kick in and not allow me to save such an entity. If I specify a value != 0 and != zero, mysql will complain for a duplicate primary key.
I'm running out of ideas and options to try. Any? (starting a bounty)

This works with eclipselink. It will create a seperate table for the sequence, but that shouldn't pose a problem.
#Id
#GeneratedValue(strategy=GenerationType.AUTO)
#Column(name="id", insertable=true, updatable=true, unique=true, nullable=false)
private Long id;
GenerationType.AUTO will choose the ideal generation strategy. Since the field is specified as insertable and updateable, a TABLE generation strategy will be used. This means eclipselink will generate another table holding the current sequence value and generate the sequence itself instead of delegating it to the database. Since the column is declared insertable, if id is null when persisting, eclipselink will generate the id. Otherwise the existing id will be used.

If you use TABLE sequencing, then EclipseLink will allow you to override the value (or SEQUENCE if your database supports this, MySQL does not).
For IDENTITY, I'm not even sure that MySQL will allow you to supply your own Id, you might want to verify this. In general I would never recommend using IDENTITY as it does not support preallocation.
There are a few issues with allowing IDENTITY to provide the id or not. One is that two different insert SQL will need to be generated depending on the id value, as for IDENTITY the id cannot be in the insert at all. You may wish to log a bug to have IDENTITY support user provided ids.
You should still be able to get it working with your own Sequence subclass, or possibly MySQLPlatform subclass. You would set your MySQLPlatform subclass using the "eclipselink.target-database" persistence unit property.

Database-centric solution to your problem:
Create an auxiliary, nullable column in your table. It will hold your manually assigned ids:
CREATE TABLE `test_table`
(
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`manual_id` bigint(20) NULL,
`some_other_field` varchar(200) NOT NULL,
PRIMARY KEY(id)
);
Map this column to a normal field in your Entity:
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
#Column(name="manual_id")
private Integer manualId;
Create a trigger that sets the table id to the manual assigned id if it is not null:
DELIMITER //
CREATE TRIGGER `test_table_bi` BEFORE INSERT ON `test_table`
FOR EACH ROW
BEGIN
IF NEW.`manual_id` IS NOT NULL THEN
SET NEW.`id` = NEW.`manual_id`;
END IF;
END;//
DELIMITER;
Always use the manualId when you need to assign a custom id. The trigger will do the magic for you:
testEntiy.setManualId(300);
entityManager.persist(testEntity);
After the database import phase, simple remove the trigger, the auxiliary column and it's mapping.
DROP TRIGGER `test_table_bi`;
ALTER TABLE `test_table` DROP COLUMN `manual_id`;
Warning
If you manually specify an id greater than the current AUTO_INCREMENT value, the next generated id will jump to the value of the manually assigned id plus 1, e.g.:
INSERT INTO `test_table` (manual_id, some_other_field) VALUES (50, 'Something');
INSERT INTO `test_table` (manual_id, some_other_field) VALUES (NULL, 'Something else');
INSERT INTO `test_table` (manual_id, some_other_field) VALUES (90, 'Something 2');
INSERT INTO `test_table` (manual_id, some_other_field) VALUES (NULL, 'Something else 2');
INSERT INTO `test_table` (manual_id, some_other_field) VALUES (40, 'Something 3');
INSERT INTO `test_table` (manual_id, some_other_field) VALUES (NULL, 'Something else 3');
Will wield the results:
+----+-----------+------------------+
| id | manual_id | some_other_field |
+----+-----------+------------------+
| 50 | 50 | Something |
| 51 | NULL | Something else |
| 90 | 90 | Something 2 |
| 91 | NULL | Something else 2 |
| 40 | 40 | Something 3 |
| 92 | NULL | Something else 3 |
+----+-----------+------------------+
To avoid problems it is highly recommended to set the AUTO_INCREMENT column to start with a number greater than all of the existing ids in your previous database, e.g.:
ALTER TABLE `test_table` AUTO_INCREMENT = 100000;

I might be missing something obvious, but why not just define another Entity with the same #Table(name=".....") annotation, with the same fields, but make the id not generated? Then you can use that Entity for the code that copies data from the old DB to the new, and the one with the generated Id can be used for normal creates that require id generation.
I can't tell you if it works with EclipseLink, but we're using Hibernate here and it doesn't seem to mind it.

Using GenerationType.SEQUENCE with PostgreSQL and EclipseLink worked for me.
1) Change
#GeneratedValue(strategy = GenerationType.IDENTITY)
by
#GeneratedValue(strategy=GenerationType.SEQUENCE, generator="step_id_seq")
#SequenceGenerator(name="step_id_seq", sequenceName="step_id_seq")
Now, you can call sequence using NativeQuery:
return ((Vector<Integer>) em.createNativeQuery("select nextval('step_id_seq')::int").getSingleResult()).get(0);
and set the returned Id to your Entity before call EntityManager.persist() method.
Hope it's not too late!

Look for Custom Id Generator
http://blog.anorakgirl.co.uk/2009/01/custom-hibernate-sequence-generator-for-id-field/
maybe this could help.

My way (MySql) is deactivate GeneratedValue:
#Id
//#GeneratedValue
#Column(unique = true, nullable = false, columnDefinition ="BINARY(16)")
private UUID id;
And add in Entity:
#PrePersist
protected void onCreation() {
if (id == null) setId(UUID.randomUUID());
}
Now in my code I can do (on service for example):
String clientID = env.getParam("id");
Entity entity = entityRepository.findFirstById(UUID.fromString(clientID));
//is new?
if (entity==null){
entity = new Entity();
entity.setId(UUID.fromString(clientID));//set cumstom ID here
}
Entity entityNew = entityRepository.save(entity); //insert
if (entityNew.getId().equals(entity.getId()) ){
Log.i("OK!")
}
👌