I encounter an optimisation problem and I can't figure out why my query is so slow.
Here my entity :
#Entity
#Table(name = "CLIENT")
public class Client {
private static final long serialVersionUID = 1L;
#Id
#Column(name = "CLIENT_ID")
#SequenceGenerator(name = "ID_GENERATOR", sequenceName = "CLIENT_S", allocationSize = 1, initialValue = 1)
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "ID_GENERATOR")
private Long id;
#Column(name="LOGIN")
private String login;
#Column(name="PASSWORD")
private String password;
And the DAO
#NoRepositoryBean
public interface ClientDao extends JpaRepository<Client, Long>, JpaSpecificationExecutor<Client> {
Client findByPasswordAndLogin(#Param("login") String customerLogin,#Param("password") String customerHashedPassword);
}
When the method findByPasswordAndLogin is executed, it takes about 200ms to be completed (seen both through Junit tests and with JProfiler).
Here the Hibernate query :
Hibernate: select clientx0_.CLIENT_ID as CLIENT_ID1_4_, clientx0_.LOGIN as LOGIN9_4_, clientx0_.PASSWORD as PASSWORD10_4_, clientx0_.STATUT as STATUT13_4_ from CLIENT clientx0_ where clientx0_.PASSWORD=? and clientx0_.LOGIN=?
When I execute manually the SQL query on the database, it takes only 3ms :
select * from CLIENT where PASSWORD='xxxxx' and LOGIN='yyyyyyyy'
We have 4000 clients in our development environnement. More than a million in production.
Here the context :
JDK 8
Spring 4.1.6.RELEASE + JPA + Hibernate
Oracle Database 10
Any idea ?
I have tested different types of DAO (I don't publish code here because it is so dirty) :
With Hibernate : ~200ms
With (Injected) Spring JDBCTemplate and RowMapper : ~70 ms
With Java Statement : ~2 ms
With Java OracleStatement : ~5 ms
With Java PreparedStatement : ~100ms
With Java PreparedStatement adjusted with Fetch size = 5000 : ~50ms
With Java OraclePreparedStatement : ~100ms
With Java OraclePreparedStatement adjusted with PreFetch size = 5000 : ~170ms
Notes :
DAO injected by Spring instead of new ClientDao() : +30ms lost (-sick-)
Connection time to DB : 46ms
I could use :
Java Statement with manual sanitized fields.
Pre-connection on application launch
Do not use Spring Injection
But :
Not really secured / safe
Fast for a small number of rows, slow to map ResultSet to entity on large number of rows (I also have this use case)
So :
The Spring JDBCTemplate with RowMapper seems to be the best solution to increase performances on specific case.
And we can keep a security on SQL queries.
But need to write specific RowMapper to transform ResultSet to Entity.
Example of Spring JDBCTemplate
#Repository
public class ClientJdbcTemplateDao {
private final Logger logger = LoggerFactory.getLogger(ClientJdbcTemplateDao.class);
private JdbcTemplate jdbcTemplate;
#Autowired
public void setDataSource(DataSource dataSource) {
this.jdbcTemplate = new JdbcTemplate(dataSource);
}
public List<Client> find() {
List<Client> c = this.jdbcTemplate.query( "SELECT login FROM Client WHERE LOGIN='xxxx' AND PASSWORD='xxx'", new ClientRowMapper());
return c;
}
}
Example of Client RowMapper
public class ClientRowMapper implements RowMapper<Client> {
#Override
public Client mapRow(ResultSet arg0, int arg1) throws SQLException {
// HERE IMPLEMENTS THE CONVERTER
// Sample :
// String login = arg0.getString("LOGIN")
// Client client = new Client(login);
// return client;
}
}
Maybe can be better, any suggestion is welcome.
Related
I'm using:
Quarkus with JPA (javax)
Postgres 11 database
I have:
An Entity
#Entity
#Table(name = "MyEntityTable")
#NamedQuery(name = MyEntity.DOES_EXIST, query = "SELECT x FROM MyEntity x WHERE x.type = :type")
public class MyEntity {
public static final String DOES_EXIST = "MyEntity.DoesExists";
#Id
#SequenceGenerator(name = "myEntitySequence", allocationSize = 1)
#GeneratedValue(generator = myEntitySequence)
private long id;
#Column(name = type)
private String type;
}
A repository
#ApplicationScoped
#Transactional(Transactional.TxType.Supports)
public class MyEntityReporitory {
#Inject
EntityManager entityManager;
#Transactional(Transactional.TxType.Required)
public void persist(final MyEntity entity) {
entityManager.persist(entiy);
}
public boolean doesExist(final String type) {
final TypedQuery<MyEntity> query = entityManager
.createNamedQuery(MyEntity.DOES_EXIST, MyEntity.class)
.setParameter("type", type);
return query.getResultList().size() > 0;
}
}
A test with two variations
Variation 1
#QuarkusTest
#QuarkusTestResource(DatabaseResource.class) // used to set up a docker container with postgres db
public class MyEntityRepositoryTest {
private static final MyEntity ENTITY = entity();
#Inject
MyEntityRepository subject;
#Test
public void testDoesExist() {
subject.persist(ENTITY);
final boolean actual = subject.doesExist("type");
assertTrue(actual);
}
#Test
public void testDoesExist_notMatching() {
subject.persist(ENTITY);
final boolean actual = subject.doesExist("another_type");
assertFalse(actual);
}
private static MyEntity entity() {
final MyEntity result = new MyEntity();
result.setType("type")
return result;
}
}
When I execute this test class (both tests) I'm getting the following Exception on the second time the persist method is called:
javax.persistence.PersistenceException: org.hibernate.PersistentObjectException: detached entity passed to persist com.mypackage.MyEntity
...
Variation 2
I removed the constant ENTITY from the test class, instead I'm calling now the entity() method inside the tests, like:
...
subject.persist(entity());
...
at both places. Now the Exeption is gone and everything is fine.
Question
Can someone explain to me, why this is the case (why variante 2 is working and variante 1 not)?
https://vladmihalcea.com/jpa-persist-and-merge/
The persist operation must be used only for new entities. From JPA perspective, an entity is new when it has never been associated with a database row, meaning that there is no table record in the database to match the entity in question.
testDoesExist executed, ENTITY saved to database and ENTITY.id set to 1
testDoesExist_notMatching executed and persist called on ENTITY shows the error beacuse it exists in the database, it has an id assigned
The simplest fix is to call entity() twice, as in you variation 2.
But don't forget that the records will exist after a test is run, and might affect your other test cases. You might want to consider cleaning up the data in an #After method or if you intend to use this entity in multiple test cases then put the perist code into a #BeforeClass method.
Is there a possibility to run some custom SQL query generated via 3rd party tools like e.g. jOOQ and still benefit from Spring Data JDBC mapping features i.e. #Column annotations? I don't want to use #Query annotation.
class Model { #Id private Long id; #Column("column") private String prop; }
class MyRepo {
public Model runQuery() {
val queryString = lib.generateSQL()
// run query as if it has been generated by Spring Data JDBC and map
// results to Model automatically
}
}
Perhaps JdbcTemplate can solve your problem? Specifically, one of the queryForObject() methods may be of interest since your are requesting a single object:
class MyRepo {
private JdbcTemplate jdbcTemplate;
#Autowired
MyRepo(JdbcTemplate jdbcTemplate) {
this.jdbcTemplate = jdbcTemplate;
}
public Model runQuery() {
val query = lib.generateQuery();
return jdbcTemplate.queryForObject(query, Model.class);
}
}
More information and other use cases can be found in the Spring Guide Accessing Relational Data using JDBC with Spring
I am new to Graphql and looking into creating a proof of concept to see how it works. I am using Spring Boot (2.2.2.RELEASE) and bringing in the graphql-spring-boot-starter.
<dependency>
<groupId>com.graphql-java-kickstart</groupId>
<artifactId>graphql-spring-boot-starter</artifactId>
<version>6.0.1</version>
</dependency>
I have setup my graphql schema as a file on the classpath with the following configuration:
type Order {
orderNumber: ID!
customers: [Customer]
items: [Item]
}
type Customer {
customerNumber: ID!
fullName: String
postalAddresses: [PostalAddress]
}
type PostalAddress {
line1: String
line2: String
city: String
stateCode: String
postalCode: String
postalCodeExtension: String
countryCode: String
}
type Item {
itemId: ID!
fullDescription: String
}
type Query {
findOrdersByCustomerNumber(customerNumber: String): [Order]
}
I have created a root Query class:
#Component
public class Query implements GraphQLQueryResolver {
private OrderService orderService;
#Autowired
public Query(OrderService orderService) {
this.orderService = orderService;
}
public List<Order> findOrdersByCustomerNumber(String customerNumber) {
return this.orderService.findOrdersByCustomerNumber(customerNumber);
}
}
And here is my Order resolver:
#Component
public class OrderResolver implements GraphQLResolver<Order> {
private ItemRepository itemRepository;
private CustomerRepository customerRepository;
#Autowired
public OrderResolver(CustomerRepository customerRepository, ItemRepository itemRepository) {
this.customerRepository = customerRepository;
this.itemRepository = itemRepository;
}
public List<Item> item(Order order) {
return itemRepository.findItemsByOrderNumber(order.getOrderNumber());
}
public List<Customer> customers(Order order) {
return customerRepository.findCustomersByOrderNumber(order.getOrderNumber());
}
}
Everything seems to work fine and I can send a graphql request and get a response. It was actually really easy to implement and impressed with how quickly this library allowed me to do that.
However, here is my issue. It's the dreaded n+1 SQL issue.
So, when I have customer with 162 orders.
1 = Driver SQL (get all the orders for the customer number)
162 = Customer SQL (One query is fired off for each order and it's same customer for each select)
162 = Postal Address SQL (One query is fired off for each customer...note, not included in the code snippets)
162 = Item SQL (assume one item per order).
So, that totals 487 SQL queries. And as a result, this has performance implications. I am using straight JDBC to query the database (no JPA or ORM's at the moment).
My questions is how can I get hold of the GraphQL request in the root Query resolver so that can manipulate the SQL for the dependent objects in the graph? When doing some research, I see that in the Node/Javacript world, there a dataloader utility that can help address this issue (https://github.com/graphql/dataloader)
So, I am unclear on how to solve this with this java implementation. If anyone has any suggestions or sample code, that would be really helpful to see if this POC has any merit.
I have found the need to limit the size of a child collection by a property in the child class.
I have the following after following this guide:
#FilterDef(name="dateFilter", parameters=#ParamDef( name="fromDate", type="date" ) )
public class SystemNode implements Serializable {
#Getter
#Setter
#Builder.Default
// "startTime" is a property in HealthHistory
#Filter(name = "dateFilter", condition = "startTime >= :fromDate")
#OneToMany(mappedBy = "system", targetEntity = HealthHistory.class, fetch = FetchType.LAZY)
private Set<HealthHistory> healthHistory = new HashSet<HealthHistory>();
public void addHealthHistory(HealthHistory health) {
this.healthHistory.add(health);
health.setSystem(this);
}
}
However, I don't really understand how to toggle this filter when using Spring Data JPA. I am fetching my parent entity like this:
public SystemNode getSystem(UUID uuid) {
return systemRepository.findByUuid(uuid)
.orElseThrow(() -> new EntityNotFoundException("Could not find system with id " + uuid));
}
And this method in turn calls the Spring supported repository interface:
public interface SystemRepository extends CrudRepository<SystemNode, UUID> {
Optional<SystemNode> findByUuid(UUID uuid);
}
How can I make this filter play nicely together with Spring? I would like to activate it programatically when I need it, not globally. There are scenarios where it would be viable to disregard the filter.
I am using Spring Boot 1.3.5.RELEASE, I cannot update this at the moment.
Update and solution
I tried the following as suggested to me in the comments above.
#Autowired
private EntityManager entityManager;
public SystemNode getSystemWithHistoryFrom(UUID uuid) {
Session session = entityManager.unwrap(Session.class);
Filter filter = session.enableFilter("dateFilter");
filter.setParameter("fromDate", new DateTime().minusHours(4).toDate());
SystemNode systemNode = systemRepository.findByUuid(uuid)
.orElseThrow(() -> new EntityNotFoundException("Could not find system with id " + uuid));
session.disableFilter("dateFilter");
return systemNode;
}
I also had the wrong type in the FilterDef annotation:
#FilterDef(name="dateFilter", parameters=#ParamDef( name="fromDate", type="timestamp" ) )
I changed from date to timestamp.
This returns the correct number of objects, verified against the database.
Thank you!
i have the following TableStructure in a PostgreSQL DB which is supposed to be the DB Backend for my WebApp:
init_db.sql
CREATE TABLE article (
id integer NOT NULL,
name character varying NOT NULL,
type_id integer NOT NULL
);
CREATE TABLE article_type (
id integer NOT NULL,
type_desc character varying NOT NULL
);
ALTER TABLE ONLY article
ADD CONSTRAINT
article_type_id_fkey FOREIGN KEY (type_id) REFERENCES article_type(id);
The basic access to this works (via DataSource Object defined in application.properties and letting Spring Boot handle the rest). I'm having now difficulties in understanding how to access/model this best in Spring Boot. Currently my Model Classes look like this:
ArticleType.java
public class ArticleType {
private Integer id;
private String name;
// Getters and Setters
}
andArticle.java
public class Article {
private Integer id;
private String name;
private String desc;
private ArticleType article_type;
// Getters and Setters
}
Following this example, i was constructing those classes:
ArticleTypeRepository.java
#Repository
public class ArticleTypeRepository {
#Autowired
protected JdbcTemplate jdbc;
public ArticleType getArticleType(int id) {
return jdbc.queryForObject("SELECT * FROM article.article_type WHERE id=?", articleTypeMapper, id);
}
private static final RowMapper<ArticleType> articleTypeMapper = new RowMapper<ArticleType>() {
public ArticleType mapRow(ResultSet rs, int rowNum) throws SQLException {
ArticleType articletype = new ArticleType();
articletype.setId(rs.getInt("id"));
articletype.setName(rs.getString("type_desc"));
return articletype;
}
};
and for the following file my question arises:ArticleRepository.java
#Repository
public class ArticleRepository {
#Autowired
protected JdbcTemplate jdbc;
public Article getArticle(int id) {
return jdbc.queryForObject("SELECT * FROM article.article WHERE id=?", articleMapper, id);
}
private static final RowMapper<Article> articleMapper = new RowMapper<Article>() {
public Article mapRow(ResultSet rs, int rowNum) throws SQLException {
Article article = new Article();
article.setId(rs.getInt("id"));
article.setName(rs.getString("name"));
// The following line is the one in question
// ArticleType at = getArticleType(Integer.parseInt(rs.getString("type_id")));
article.setArticle_type(at);
article.setDesc(rs.getString("description"));
return article;
}
};
What is the best practice to get the ArticleType here for the Article? Is this anyway good practice to retrieve those objects? Or should I just use a plain String object in the Article Object and query this with a view or something? I looked through the internet for "Spring Boot JDBC Nested Object Java Access Modeling" and the alike, but couldn't find any real hints or tutorials to this specific question, which makes me wonder if i'm doing something conceptually completely wrong. Any hints are appreciated (tutorials, doc's, paradigms how to do this properly, etc.)
I'll double post M. Deinum 's answer here, since it got me rolling until i switched to Hibernate/JPA:
By creating a query that returns everything you need. Write a select
statement that joins both tables.