Develop Reference

Hibernate not batching inserts

I hope you can help me with this problem since I'm really clueless by now and none of the related questions could help me so far.
I have a collection of entities I want to batch insert into TimescaleDB (extension of Postgresql) via spring-data saveAll() and I thought I configured everything by the book, but the hibernate stats never reflect a batch insert:
1223268 nanoseconds spent acquiring 1 JDBC connections;
0 nanoseconds spent releasing 0 JDBC connections;
34076442411 nanoseconds spent executing 3408 JDBC statements;
0 nanoseconds spent executing 0 JDBC batches;
My Hibernate properties are configured via HibernatePropertiesCustomizer like this:
props.put("hibernate.generate_statistics", true);
props.put("hibernate.order_inserts", true);
props.put("hibernate.order_updates", true);
props.put("hibernate.jdbc.batch_size", "100");
I validated that a transaction context is indeed present during saveAll() by calling:
TransactionSynchronizationManager.isSynchronizationActive()
And the entity's ID looks like this. There are no #GeneratedValue or #SequenceGenerator annotations, since I use the data's timestamp (from a web API) as the ID.
#Id
#Column
private Instant time;
I even tried by adapting my connection String to control the rewrite behavior, but it did not help:
jdbc:postgresql://localhost:5432/db?reWriteBatchedInserts=true&currentSchema=finance-data
I tested this with both maven org.postgresql 42.2.9 and 42.2.13 drivers.
The Spring Boot release train version is the 2.2.2.RELEASE which bundles hibernate-core 5.4.9.FINAL and spring-jdbc 5.2.2.RELEASE.
The timescale docker version is 1.7.1-pg12.
Please let me know if you need any more additional information. Thanks in advance!!!

From this, it mentions if the entity being batched inserted is manually assigned its ID , you have to add a #Version property. But for PostgreSQL , adding #Version is not required if using SEQUENCE generator to generate the ID.
I do not try if adding #Version can solve the problem if the ID is manually assigned but you could have a try. I personally use SEQUENCE generator to generate ID and it works without adding #Version in PostgreSQL.
And in order to generate ID effectively when using SEQUENCE generator , I also changed to use "pooled" or "pooled-lo" algorithm that is mentioned in this to reduce the database round trip to get the ID.
Here are what I do :
#Entity
public class Foo {
#Id
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator="foo_sequence")
#SequenceGenerator(name="foo_sequence", sequenceName = "foo_id_seq", allocationSize = 100)
private Long id;
}
And the hibernate setting :
hibernate.order_inserts = true
hibernate.order_updates = true
hibernate.jdbc.batch_size = 50
hibernate.jdbc.batch_versioned_data = true
# For using "pool-lo" optimiser for generating ID when using JPA #SequenceGenerator
hibernate.id.optimizer.pooled.preferred = pooled-lo
And also need to make sure the sequence in PostreSQL is aligned with what is configured in #SequenceGenerator :
alter sequence foo_id_seq increment by 100;
For completeness , in case of PostgreSQL , also add reWriteBatchedInserts=true in the JDBC connection string which can provides 2-3x performance improvement which is said by the docs.

Springboot 2 CrudRepository.save always throws ConstraintViolationException

I have Migrated the Springboot version from 1.4.3.RELEASE to 2.1.0.RELEASE in my project. After that CrudRepository.save() always throws org.hibernate.exception.ConstraintViolationException: could not execute statement.
This is what I can see in the Logs :
o.h.e.j.s.SqlExceptionHelper[m: SQL Error: 1062, SQLState: 23000
o.h.e.j.s.SqlExceptionHelper[m: Duplicate entry '11' for key 'PRIMARY'
o.h.i.ExceptionMapperStandardImpl[m: HHH000346: Error during managed flush [org.hibernate.exception.ConstraintViolationException: could not execute statement
This is the entity I am trying to save.
#Getter
#Setter
#Entity
#Table(name = "project_m")
public class Project {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
#Column(name = "id", nullable = false)
private Long id;
#Column(name = "name" , nullable = false)
private String name;
//other fields
}

What has changed from Springboot 1.4.3.RELEASE to 2.1.0.RELEASE is the internal Hibernate version from 5.0 to 5.3.
And what has changed in this is the way the SequenceGenerator works, which is used in case the strategy is GenerationType.AUTO (as is in your case).
Link to hibernate migration doc here.
More details on hibernate generation strategy here.
My guess is there are 2 parallel sessions inserting into this table, and both of them now share a local copy of the sequence number, which creates this collision. Not sure though!
My suggestion would be to change the strategy to GenerationType.SEQUENCE and try.

ConstraintViolationException occurred due to the fact that your primary key constraint for SQL database violates
In SQL the primary key is a unique key to identify the record, the database will throw an exception when you trying to insert a duplicate value to a primary column.
which in turn got by the hibernate and passed to your code and is the reason for this exception.
From Springboot 1.4.3.RELEASE to 2.1.0.RELEASE Hibernate version is updated from 5.0 to 5.3.
The way Hibernate interprets AUTO generation type has changed starting with Hibernate version 5.0
If you use strategy="AUTO", Hibernate will generate a table called hibernate_sequence to provide the next number for the ID sequence. You may have forgotten to add the AutoIncrement feature to your table's PK.
Another way to fix is use following annotations with strategy="AUTO"
#Id
#GeneratedValue(
strategy= GenerationType.AUTO,
generator="native"
)
#GenericGenerator(
name = "native",
strategy = "native"
)
private Long id;
You may use generation strategy strategy="IDENTITY" to enforce using the AutoIncrement feature available in SQL and avoid creating a table.
Pls check here to get some more insights

I got a workaround for this issue by setting a large value in hibernate_sequence table. I saw that the primary key value from the duplicate primary key error is generated from a table named hibernate_sequence.
When we set GenerationType.AUTO in the entity,Hibernate selects the generation strategy based on the Hibernate dialect. In older versions , Hibernate selected the GenerationType.IDENTITY as default for MySQL databases.
It now selects the GenerationType.TABLE which uses a database table to generate primary keys

Where to set hibernate.id.new_generator_mappings property?

I've wasted too much time on this ...
I'm using oracle and I have a sequence (MY_TABLE_SEQ) defined which increments by 1.
In my Pojo I have:
#SequenceGenerator(name = "MY_SEQ", sequenceName="MY_TABLE_SEQ", allocationSize=50)
#GeneratedValue(strategy=GenerationType.SEQUENCE, generator="MY_SEQ")
This gives me a unique constraint issue. From my understanding I need to set the following property:
hibernate.id.new_generator_mappings=true
I've tried setting in my hibernate.cfg.xml file but it does not seem to make any difference. I've come across server post to place in persistance.xml but this is a standalone app, no webcontainer.
Setting allocationSize=1 works but of course it hits the db on each insert to get the next sequence. Setting the above property is suppose to resolve it.

I haven't tried Oracle, but I had similar issues to yours inserting into an AS400 DB2 table.
I had to remove the identity flag on the id column on DB2 table - and instead used a custom jpa/hibernate sequence generator. This is set up on the pojo/entity annotation of the #ID entity field as you've done.
DB2 had been giving me errors about missing SYSIBM.SYSSEQUENCES table, so evidently hibernate (version 5.2), doesn't recognize the native DB2 identity designation. A custom sequence was and effective workaround.
On the #ID entity field:
#GeneratedValue(generator = "table", strategy=GenerationType.TABLE)
#TableGenerator(name = "table", allocationSize = 20)
This example allocates a pool of 20 sequence numbers each time it queries the table.
Next, create the required table Hibernate needs with columns that match the hibernate5 API - must be in lower case ... so put quotes around the names to work around the auto-upper casing that DB2 defaults to. The API will error out if these names are in caps.
Table:
"hibernate_sequences"
example of 2 Columns used:
"sequence_next_hi_value" (integer, not nullable, 0 default)
"sequence_name" (character, sample length 20, not nullable, natural default)
In the configuration code for the dialect used - ex: Spring Boot programmatically, add these properties:
properties.put("hibernate.supportsSequences","false");
properties.put("hibernate.id.new_generator_mappings","false");
and in the *.properties file:
spring.jpa.properties.hibernate.dialect.supportsSequences=false
spring.jpa.properties.hibernate.id.new_generator_mappings=false
Database systems are case-sensitive for schema/table/field names. Also watch for typos everywhere, incl. property names.
Be sure your pojo/entity only contains private fields that will be mapped to the table. Static finals such as serialVersionUID are ok.
I will be doing someething similar for SQL Server soon.
For MySQL, I had no issues using an identity column as defined in a table ID field to insert records, so didn't have to make all these changes. A simpler setup since hibernate recognizes the identity designation in MySQL.
#GeneratedValue(strategy=GenerationType.IDENTITY)
was all that was needed in the pojo.
I'm a newbie at all this, so always looking for better ways ... but this worked for now.

I set the property like this in the hibernate.cfg.xml file and it works !
<property name="hibernate.jpa.compliance.global_id_generators" value="true"/>

Hibernate JPA Column "NEXTVAL" not found

We changed the JPA provider from EclipseLink to Hibernate 4.1.3. A problem occured when attempting to persist entity using generated id. This gives the following error:
JdbcSQLException: Column "NEXTVAL" not found; SQL statement: select nextval for SCHEMA.ID_SEQ_GENERATOR
The sequence for the entity looks like the following:
#Id
#SequenceGenerator(name = "ID_GENERATOR", initialValue = 100000, allocationSize = 1, sequenceName = "ID_SEQ_GENERATOR")
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "ID_GENERATOR")
#Column(name = "ID", nullable = false)
public Long getId() {
return this.id;
}
I tried changing the 'strategy = GenerationType.IDENTITY', but it had no effect.
Any advice?

I found out that there are different databases used depending on the environment used. The dialect used is (currently) always the same even though it should change depending on the server the code is executed on. (The correct dialect is actually loaded first, but it then gets replaced because of spring decides to use another class annotated with #Configuration. I think that using profiles should get me past this, though still working with this...)
I found this link which helped me to get on right tracks:
http://techmagik.blogspot.fi/2012/07/sequence-support-in-jpa-for-db2-zos.html.html
Discovering that the H2 was used, I changed the SQL to :
"NEXT VALUE FOR " + sequenceName
Which fixed the error
So the actual problem is with loading correct dialect (as 'Luca Basso Ricci' hinted). The dialect is for DB2 (always), when it should be for H2 in some cases. For some reason this problem manifested when the provider was changed so it made it a bit difficult to track right reason.

Auto generate unique random string in Spring MVC + Hibernate

Background
I'm writing a project using Spring MVC (Framework v4.0.6.RELEASE, JPA v1.6.2.RELEASE) and Hibernate (Core v4.3.6.FINAL, JPA API v2.1). In my project, there are entities called 'Project'. Each of these projects have their unique, auto-generated IDs as primary keys. This ID is generated by the following code:
#Id
#Column(name = "project_id")
#GeneratedValue(strategy = GenerationType.AUTO)
private Long projectId;
This code works as expected and automatically creates unique IDs.
Problem
Each of these projects are supposed to have a random, unique1 'secret' String, just like those assigned by API providers like Facebook, Twitter, etc. So, to achieve this, I tried using the following code, as per the Hibernate docs:
#Column(name = "project_secret", nullable = false, unique = true)
#GenericGenerator(name = "uuid-gen", strategy = "uuid")
#GeneratedValue(generator = "uuid-gen")
private String projectSecret;
However, whenever I try to create a new project entity, I'm greeted by a org.springframework.dao.DataIntegrityViolationException with root cause:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Column 'project_secret' cannot be null
This should be auto-generated by Hibernate on creation, must be random and unique1. A 128-bit UUID is enough for me (32 characters w/out dashes) and I read that Hibernate has a UUID generator, so that's what I was aiming to use.
Further Info
After searching for hours, I'm no closer to solving it the way I want to do it. I found one possible solution, which is to include:
#PrePersist
private void generateSecret(){
this.setProjectSecret(UUID.randomUUID().toString());
}
in the Project entity class. When this method is inserted (and #GenericGenerator & #GeneratedValue tags removed), the project secret is correctly generated and inserted; system works as expected; no exceptions are thrown. However, (I believe) this can't ensure uniqueness2 and just causes an exception when a duplicate secret is inserted. I want to ensure uniqueness and preferably want to solve this with built-in Hibernate generators.
(Notes)
I actually am not sure if uniqueness should be enforced. I suppose having every secret unique can (theoretically) create an additional layer of security, which takes me to:
I realise that the probability of UUID collision is very very low, so UUID generation ensures that uniqueness in a probabilistic sense but can (or should) I be really sure of it?

I had an issue like this before and I realised after a while that it was my database table that was causing the issue. This might be the same problem you are having...
For your project_id ensure you use the following when you are creating that column in the database
GENERATED ALWAYS AS IDENTITY
I hope this is the same issue and that this will be of help to you. Also would recommend using uuid2 as your strategy.
see here... http://docs.jboss.org/hibernate/core/3.6/reference/en-US/html/mapping.html#d0e5294
Edit
After realising that project_secret is not the #id field then the answer is that hibernate does not support generated values on any column except for the #id field.
See here for more details : Hibernate JPA Sequence (non-Id)