We have a bog standard REST-based Hibernate application.
Recently we noticed that it dies the death by Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded time and again. We've emulated what happens there in a test, that collects items from a collection of elements. After about 1000 runs the application goes kaput with the pattern as below. The production displays similar behavior, although the time scale is much longer:
Java profiler shows the abundance of hibernate Node elements in a hashmap:
When drilling deeper into the what these nodes are I can see the following:
It looks like Hibernate caches HQL queries (even if we run the same query over and over again).
The class that comprises the offending collection is below:
#Entity
#Audited
#Table(name = "benefit_package", uniqueConstraints = {
#UniqueConstraint(name = "uk01_benefit_package", columnNames = {"employer_guid", "benefit_package_name"}),
#UniqueConstraint(name = "uk02_benefit_package", columnNames = {"benefit_package_guid"})})
#SequenceGenerator(name = "benefit_package_sequence", sequenceName = "benefit_package_id_seq", allocationSize = 1, initialValue = 1)
#EqualsAndHashCode(of = {"employerId", "name"})
#ToString(of = {"id", "employerId", "name"})
public class BenefitPackage {
#Id
#Column(name = "benefit_package_id")
#GeneratedValue(generator = "benefit_package_sequence", strategy = SEQUENCE)
private Long id;
#Column(name = "benefit_package_guid", nullable = false, columnDefinition = "binary(16)")
private UUID guid;
#Column(name = "benefit_package_name", nullable = false)
private String name;
#Column(name = "employer_guid", nullable = false, columnDefinition = "binary(16)")
private UUID employerId;
#ManyToOne
#JoinColumn(name = "selection_control_id")
private SelectionControl selectionControl;
#OneToMany(mappedBy = "benefitPackage")
private List<BenefitPackageVersion> versions = new ArrayList<>();
#Column(name = "last_modified_date")
private LocalDateTime lastModifiedDate;
#Version
#Column(name = "optlock")
private Long optLock;
This entity is used with pagination, but we do not use any custom JPA queries (at least overtly) for this operation.
Application details: SpringBoot 1.2.8, Hibernate 5.1 (Upgrading to 5.2.18Final had no effect). No second level cache is used.
Questions:
Why should Hibernate cache the same query over and over again?
Has this ever been noticed or addressed? Is there a fix for this problem?
I know the ticket might be a duplication, but I have not found definitive answer to that question anywhere.
Related
I'm getting a weird error while I'm debugging my POC.
I have 2 entities:
#Getter
#Setter
#Entity
#Table(name = "APPLICANT")
public class Applicant implements Serializable {
private static final long serialVersionUID = 6060170457948717553L;
#Id
#Column(name = "applicant_id", insertable = false, nullable = false)
private Long applicantId;
#Column(name = "application_id", unique = true)
private String applicationId;
#OneToOne(fetch = FetchType.LAZY, optional = false)
#JoinColumn(name = "application_id", referencedColumnName = "application_id", insertable =
false, updatable = false)
private ApplicationEntity applicationEntity;
#Getter
#Setter
#Entity
#Table(name = "APPLICATION")
public class ApplicationEntity implements Serializable{
private static final long serialVersionUID = 7300036359295729197L;
#Id
#Column(name = "APPLICATION_ID")
private String id;
#OneToOne(mappedBy = "applicationEntity", fetch = FetchType.LAZY)
private Applicant applicant;
These classes has the repositories interfaces extending from CrudRepository, and in the Applicant repository I have a custom method to get the entity with the applicationId:
Applicant findByApplicationId(String applicationId);
But, when I'm debugging, I see the following message in the intellij debuguer on the applicationEntity attribute:
Method threw 'org.hibernate.LazyInitializationException' exception. Cannot evaluate org.example.postgres.jpa.model.ApplicationEntity$HibernateProxy$qa4PKx8V.toString()
The value qa4PKx8V changes every time that I perform a new test.
I tried a lot of combinations in the #Join annotation, I've deleted the lombook annotations, I've used the #Transactional annotation either, but is always the same error.
A key point to note, is that I can get the data from the table with any error, I just see this message in the debugger, so my question is, this is a thing of intellij or something like that? Or I need to fix this with configuration or changing something in my code?
Thanks.
I am assuming you have an autogenerated toString() implementation?
In general, you should avoid referencing lazily-loaded properties in toString(), equals(), hashCode() etc. Failing to do so will result in LazyInitializationException surprises like the one you're facing, triggered by the aforementioned methods whenever they try to access lazy properties outside of an active transaction context.
(This is indeed 'a thing of intellij', in the sense that although the debugged code is probably surrounded by a transaction, the Intellij inspector evaluates the expression on a separate thread where no transaction is active = no persistence context is open. Also, it will only happen with #XxxToOne(optional = false) properties)
Say I have at least two entities.
#Entity
public class Process {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
#Column(unique = true)
private String name;
#ManyToAny(
metaColumn = #Column(name = "node_type"),
fetch = FetchType.LAZY
)
#AnyMetaDef(
idType = "long", metaType = "string",
metaValues = {
#MetaValue(targetEntity = Milestone.class, value = MILESTONE_DISC),
#MetaValue(targetEntity = Phase.class, value = PHASE_DISC)
}
)
#Cascade({org.hibernate.annotations.CascadeType.ALL})
#JoinTable(
name = "process_nodes",
joinColumns = #JoinColumn(name = "process_id", nullable = false),
inverseJoinColumns = #JoinColumn(name = "node_id", nullable = false)
)
private Collection<ProcessNode> nodes = new ArrayList<>();
...
}
#Entity
#ToString
#DiscriminatorValue(MILESTONE_DISC)
public class Milestone implements ProcessNode {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
private String name;
#OneToMany(cascade = CascadeType.ALL, fetch = FetchType.LAZY)
private Collection<ResultDefinition> results;
#ToString.Exclude
#ManyToOne(fetch = FetchType.LAZY)
#Transient
private Process process;
...
}
Now I want to use spring data jpa specification to find (all) processes which have a milestone with name "S5".
Note that Milestone is a ProcessNode and there is another Entity called Phase which is also a ProcessNode. These can be contained in the "nodes" collection of my Process Entity.
I tried to write something like this:
public static Specification<Process> hasMilestoneWithName(final String milestoneName) {
return (Specification<Process>) (root, query, criteriaBuilder) -> {
Path<?> namePath = root.join("nodes").get("name");
return criteriaBuilder.equal(namePath, milestoneName);
};
}
This does not work, but throws:
Caused by: java.lang.IllegalArgumentException: Unable to locate Attribute with the the given name [nodes] on this ManagedType [com.smatrics.dffs.processservice.model.entities.Process]
I don't really know how to use the API. Examples often refer to a meta-model that would be generated by the IDE or maven, but I really do not want to have any static generated resources. Please help me resolve this with Specification of spring-data-jpa without a generated meta-model.
Also if you could help me write the hql it would be awesome.
Thanks!
I would suggest a simpler alternative, coming from bottom-up:
Load Milestone entities with name=S5: findByName("S5")
Return the Process for each Milestone
Filter out the duplicates
Or you could even save a few SQL queries by returning not the Milestone entity but only the ID of the Process for each Milestone and then load the Process nodes by a list of IDs:
The (native) SQL equivalent would be
select *
from process
where id in (
select process_id
from milestone
where name = 'S5'
)
Regardless of my solution your join does not look completely correct to me but I can't point out what's wrong - maybe there are other methods on the JPA metamodel that return a CollectionJoin? Not sure. Probably it is because #ManyToAny is not JPA standard so the JPA criteria API does not recognize nodes as a valid "joinable" field.
I am running a spring boot application running Hibernate 5.4.2 and Postgresql 42.2.2 DB running on Heroku.
I am using JPARepository's default findAll() method to select my entire table. There are about ~50 entities and it's taking ~35 seconds. When I run a select all query using the Heroku CLI via terminal on the same machine, the query finishes in ~130milliseconds.
I have been trying to use JProfiler to pinpoint the problem. Looking at the query via JProfiler, I notice that the associated JDBC call takes about ~140ms which is close to the Heroku CLI. However the query itself still takes 35 seconds...
I have turned hibernate logging on and made sure that there are no other queries being sent. Also, I have tried to run the same query backing my application with a H2 database. The query returned in less than a hundred milliseconds.
See JProfiler query time https://i.stack.imgur.com/tAKwf.png
My entity is quite simple. There are no relationships to other classes so I dont think its a n+1 query problem.
#Entity
#Table(name = "event")
#NoArgsConstructor
#Getter
#ToString
#EqualsAndHashCode(callSuper = true)
public class SqlEvent extends SerializedObjectRecord {
#Id
#GeneratedValue(strategy = GenerationType.SEQUENCE)
#Column(name = "id", updatable = false, nullable = false)
#NonNull
private Long id;
#Column(nullable = false)
#NonNull
private String name;
#Column(nullable = false)
#NonNull
private UUID uuid;
#Column(nullable = false)
#NonNull
private UUID sqlCommandUuid;
#Column(nullable = false)
#NonNull
private Instant timestamp;
#Column(nullable = false)
#NonNull
private String avroSchemaFingerprint;
My service class
#Override
#Timed(value = "service")
public Collection<Pair> getAll(UUID commandId) {
long start = System.nanoTime();
List<SqlEvent> list = this.sqlEventRepository.findAll();
long end = System.nanoTime();
long duration = (end - start) / 1000000; #duration takes ~35000ms like Jprofiler says
return convert(list);
}
[1]: https://i.stack.imgur.com/tAKwf.png
I am trying to get familiar with spring and jpa. For start there is a table anime_details containing details of an anime. As it can have many genres, db has another table named genre. The intermediate table to contain their many to many relationship entries is also there. When I query for any anime by id, it should return the details of the anime along with the genres.
It does return an object with details of the anime and list of Genre objects (which is as expected). But what I want is to restrict the columns that will be fetched from Genre objects. For example only id or just id and name (In case there are more columns other than these).
AnimeDetails
#Getter
#Setter
#Entity
#Table(name = "anime_details")
public class AnimeDetails {
#Id
#SequenceGenerator(name = "animeDetailsSeq", sequenceName =
"anime_details_id_seq",
allocationSize = 1)
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator =
"animeDetailsSeq")
private Integer id;
private String name;
private Integer episodes;
private Date started;
private Date ended;
private String status;
#ManyToMany
#JoinTable(
name = "anime_genre",
joinColumns = #JoinColumn(name = "details_id", referencedColumnName = "id"),
inverseJoinColumns = #JoinColumn(name = "genre_id", referencedColumnName = "id"))
#JsonManagedReference
private List<Genre> genres;
protected AnimeDetails() {
}
}
Genre
#Data
#Entity
#Table(name = "genre")
public class Genre {
#Id
#SequenceGenerator(name = "genreSeq", sequenceName = "genre_id_seq",
allocationSize = 1)
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "genreSeq")
private Integer id;
private String name;
#ManyToMany(mappedBy = "genres")
List<AnimeDetails> animes;
protected Genre() {
}
}
Expected payload
{
"id": 2,
"name": "Your Name",
"episodes": 1,
"started": "2016-08-25T18:00:00.000+0000",
"ended": "2016-08-25T18:00:00.000+0000",
"status": "Completed",
"genres": [
{
"id": 5,
"name": "Drama"
},
{
"id": 10,
"name": "Supernatural"
}
]
}
Right now, I get the result and manually get columns one by one and set those in a DTO. But that is not efficient as the database query is already fetching more data than needed.
Is there any specific annotation/property/jpql to reduce it?
Indeed was looking for a proper solution regarding the same issue , cause as you pointed out it is creating performance issues as there are huge useless data loads between the APP and the DB. Imagine that there could be, not only one query but much more and you need a global optimization solution...
From the first place Spring DATA is not supporting this operation so its leading you at the manual configuration and set up on a DTO reference. The same applies if you were using a custom Object and returning that inside the JPQL with the constructor trick , or else write a native query , get back a List<Object> and again manually map the data back to your actual object , which is the most efficient but not elegant solution
More info in this link , but try to check both answers for the details.
The other thing is that as you are using hibernate underneath , which is providing custom mappers , you could always write up your custom HQL(not jpql) , set up a proper DAO , wire up the EntityManager or directly the SessionFactory (which is breaking the abstract JPA contract , but you can utilize the full goodies that hibernates offers) and then return the same object, but only with the columns you need.
Example for the second point:
import javax.persistence.EntityManager;
import org.hibernate.query.Query;
import org.hibernate.transform.Transformers;
public CustomEntity getEntity(){
Query<CustomEntity> q = (Query<CustomEntity>) entityManager.createQuery("select
e.id,e.name from CustomEntity e where e.name = 'parameter'");
q.setResultTransformer(Transformers.aliasToBean(CustomEntity.class));
CustomEntity entity = (CustomEntity) q.getSingleResult();
return name;
}
Note CustomEntity is a managed Entity Bean / Table in the database , just placing this example to be close on what you might need to achieve.
Tried with
Spring boot 2.0.5.RELEASE
Spring Data 2.0.5.RELEASE
Hibernate-core 5.2.17.Final
I tried a different solution today. Lets look at the code first.
Genre
#Data
#Entity
#Table(name = "genre")
public class Genre {
#Id
#SequenceGenerator(name = "genreSeq", sequenceName = "genre_id_seq",
allocationSize = 1)
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "genreSeq")
private Integer id;
private String name;
#ManyToMany
#JoinTable(
name = "anime_genre",
joinColumns = #JoinColumn(name = "genre_id", referencedColumnName = "id"),
inverseJoinColumns = #JoinColumn(name = "details_id", referencedColumnName = "id"))
List<AnimeIdentity> animes;
protected Genre() {
}
}
AnimeIdentity
#Getter
#Setter
#Entity
#Table(name = "anime_details")
public class AnimeIdentity {
#Id
#SequenceGenerator(name = "animeDetailsSeq", sequenceName =
"anime_details_id_seq",
allocationSize = 1)
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator =
"animeDetailsSeq")
private Integer id;
private String name;
protected AnimeIdentity() {};
}
Queries Hibernate made
Hibernate: select genre0_.id as id1_2_0_, genre0_.name as name2_2_0_ from genre genre0_ where genre0_.id=?<br>
Hibernate: select animes0_.genre_id as genre_id2_1_0_, animes0_.details_id as details_1_1_0_, animeident1_.id as id1_0_1_, animeident1_.name as name2_0_1_ from anime_genre animes0_ inner join anime_details animeident1_ on animes0_.details_id=animeident1_.id where animes0_.genre_id=?
Feel free to show me the pros and cons of this solution. To me its a good solution if my necessity is limited to only this. But in case of different type of queries making more and more entity pojos will be a tiresome task.
I am attempting to get Hibernate to lazy load some clobs. The loading portion is working just fine. The issue is when I try to create a new one. I started with advice from Blob lazy loading
Here are my mappings (Note the table structure is really really bad, there are multiple clobs on this table -- this example is simplified from my real model...).
#Entity #Table("TABLE_1")
public class BadDBDesign {
#Id #GeneratedValue(strategy = GenerationType.IDENTITY)
#Column("table_id")
private long key;
#OneToOne(fetch = FetchType.LAZY, cascade = CascadeType.ALL)
#JoinColumn(name = "table_id", referencedColumnName = "table_id",
insertable = true, updatable = false)
private BlobWrapperA;
#OneToOne(fetch = FetchType.LAZY, cascade = CascadeType.ALL)
#JoinColumn(name = "table_id", referencedColumnName = "table_id",
insertable = true, updatable = false)
private BlobWrapperB;
}
#Entity #Table(name = "TABLE_1")
public class BlobWrapperA {
#Lob
#Column(name = "col_A", nullable = false)
#Type(type = "org.springframework.orm.hibernate3.support.BlobByteArrayType")
private byte[] blobColA;
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name = "table_id")
private long Key;
}
#Entity #Table(name = "TABLE_1")
public class BlobWrapperB {
#Lob
#Column(name = "col_B", nullable = false)
#Type(type = "org.springframework.orm.hibernate3.support.BlobByteArrayType")
private byte[] blobColB;
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name = "table_id")
private long Key;
}
Application boots just fine, am able to retrieve the data without loading the clobs (am able to retrieve them when needed via lazy loading), but when I attempt to create the new ones I receive the following stacktrace:
Hibernate:
insert
into
TABLE_1
(key, col_A, col_B)
values
(?, ?, ?)
2011-08-31 17:35:09,089 [http-8080-1] DEBUG org.springframework.jdbc.support.lob.DefaultLobHandler IP134.167.141.34 CV#f2a597b2-a185-4e89 P#71252 - Set bytes for BLOB with length 7136
2011-08-31 17:35:16,441 [http-8080-1] DEBUG org.springframework.jdbc.support.lob.DefaultLobHandler IP134.167.141.34 CV#f2a597b2-a185-4e89 P#71252 - Set bytes for BLOB with length 10946
Aug 31, 2011 5:35:50 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet online threw exception java.sql.SQLIntegrityConstraintViolationException: ORA-01400: cannot insert NULL into ("SCHEMA"."TABLE_1"."COL_A")
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:837)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1010)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1315)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3576)
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3657)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1350)
at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
Note the important piece where we see the lengths of the clobs immediately after the Hibernate insert statement from the generated SQL.
Edit: After looking at this early this morning, I realized that the issue was due to one of the Blobs had to be mapped with #JoinColumn(insertable = false, updatable = false), otherwise Hibernate would not start. As such of course it was attempting to insert Null into this column. So the new question becomes, can you lazily MULTIPLE clobs on a single table (using the same key). I'm guessing without a table redesign, I'm pretty much out of luck unless Oracle fixes the driver.
As much as it makes me want to vomit we needed to get this functionality without modifying the Database.
As such, I pulled out the common pieces into an Abstract class like such:
#MappedSuperclass #Table("TABLE_1")
public class BadDBDesign {
#Id #GeneratedValue(strategy = GenerationType.IDENTITY)
#Column("table_id")
private long key;
#Column("small_value")
private String smallVarChar2Field;
}
The problem is I then have to extend this class for each of our blobs :( Thus our extended classes loook like:
public class BlobA extends BadDBDesign {
#Lob #Column("col_a")
#Type(type ="org.springframework.orm.hibernate3.support.BlobByteArrayType")
private byte[] blobColA;
}
public class BlobB extends BadDBDesign {
#Lob #Column("col_b")
#Type(type ="org.springframework.orm.hibernate3.support.BlobByteArrayType")
private byte[] blobColB;
}
Luckily we don't have any location where we more than one clob on any given page. This is still a maintenance nightmare, but was an acceptable trade-off (for the time-being) on getting the loads done more efficiently. I created DAO's for these, which the project didn't have prior; hopefully this will push the team in a good direction towards a proper abstraction layer, and we can hopefully completely remove these wasted POJOs in a future release.
Looks like in your BlobWrapperA class you have "nullable = false" set on that column. Or the column has a null constraint on the table itself in the database.
Oracle and Hibernate hate each other when it comes to LOB types, which stems from the fact that the Oracle driver is garbage. I believe I've run across this before, you should try setting the following system properties:
hibernate.jdbc.use_streams_for_binary=true
hibernate.jdbc.batch_size=0