I am trying to rewrite this query to improve performance given that it takes more than 10 mins to execute. I believe the issue is mainly due to the large Inbox table. As union isn't an option in HQL, what can I do to improve this performance? Thanks in advance.
select distinct p from PrReq p, Inbox i where i.documentNo in
(select distinct kdn.kdnId from PrKdn kdn where kdn.prReqId = p ) or
i.documentNo in (select distinct grn.grnId from PrGrn grn where
grn.prReqId = p) and i.inboxStatus = 0 and i.moduleStatus =
:currentStatus and p.recvDept.code = :dept and p.organization = :org
order by p.billToDept, p.prId ASC, p.createDate desc
Related
I have a problem about sql performance, my db is too many rows, so this make long time to query.
SELECT * FROM A JOIN B ON A.id = B.id where ...
So I change to
SELECT * FROM A where A= a...
SELECT * FROM B where B= b...
I got 2 resultSet from 2 query here.
Can someone help me how to join 2 resultset with the best performance.
I have to split to 2 query because this database have 10 mil records.
Select col1, col2 ...
from
(
-- first query
) as tab1
join
(
-- second query
) as tab2 on tab1.colx = tab2.coly
I'm using a query in my repository and i'm trying to select 10 random questions from a table. but I get this error
ASC or DESC expected got '('
and i have no idea why
#Query ( value = "select q from Question q where q.chapitre_id=:id order by
random() limit 10", nativeQuery = true)
List<Question> findQuestionsByChapitre(#Param("id") Long id);
First of all, ordering by 1, 2, 3 or whatever number means ordering by the column in the corresponding position; if you only select one column (q in your case) you will be able to order only by that (it may work in older versions though).
Then, the correct syntax for ordering by a casual value is order by rand()
This is what worked for me :
#Query ( value = "select * from question q where q.chapitre_id=:id order
by rand() limit 10", nativeQuery = true)
Question[] findQuestionsByChapitre(#Param("id") Long id);
I am using queryDSL JPA.
I am trying to load data in chunks, so I am using .offset() and .limit()
I am querying 'single_entry' table, that contains 10000 records.
I use the following function to query data
private Collection<SingleEntry> getSingleEntries(FiscalPeriod fp, Predicate filter, Integer offset, Integer limit) {
BooleanExpression initialFilter = getSingleEntriesFilter(fp);
return query.selectFrom(QSingleEntry.singleEntry).where(initialFilter.and(filter)).offset(offset).limit(limit).fetch();
}
private BooleanExpression getSingleEntriesFilter(FiscalPeriod fp) {
int period = fp.getPeriod();
int year = fp.getYear();
QSingleEntryModification sem = QSingleEntryModification.singleEntryModification;
QSingleEntry se = QSingleEntry.singleEntry;
BooleanExpression fiscalPeriodMatches = se.year.eq(year).and(se.period.loe(period));
//excludes single entries that moved from this period
BooleanExpression excludeMoved = se.documentNumber.notIn(query.selectDistinct(sem.documentNumber).from(sem)
.where(sem.moved.eq(true)
.and(sem.originYear.eq(year)
.and(sem.originPeriod.loe(period)).and(sem.movedToPeriod.gt(period).or(sem.movedToYear.ne(year))))));
BooleanExpression movedToPeriodOrNonMoved = se.documentNumber.in(query.selectDistinct(sem.documentNumber).from(sem)
.where(sem.moved.eq(false).and(sem.originYear.eq(year).and(sem.originPeriod.loe(period))).or(sem.movedToPeriod.loe(period).and(sem.movedToYear.eq(year)))));
return fiscalPeriodMatches.and(excludeMoved).or(movedToPeriodOrNonMoved);
}
Hibernate produces the following query when offset equals 0 and limit are set:
select
* //replaced with * to be concise
from
single_entry singleentr0_
where
(
singleentr0_.year=?
and singleentr0_.period<=?
and (
singleentr0_.document_number not in (
select
distinct TOP(?) singleentr1_.document_number
from
single_entry_modification singleentr1_
where
singleentr1_.moved=?
and singleentr1_.origin_year=?
and singleentr1_.origin_period<=?
and (
singleentr1_.moved_to_period>?
or singleentr1_.moved_to_year<>?
)
)
)
or singleentr0_.document_number in (
select
distinct singleentr2_.document_number
from
single_entry_modification singleentr2_
where
singleentr2_.moved=?
and singleentr2_.origin_year=?
and singleentr2_.origin_period<=?
or singleentr2_.moved_to_period<=?
and singleentr2_.moved_to_year=?
)
)
and singleentr0_.account_number=?
This query return one result, however, if I run this query in a database manager, it returns a correct number of results.
If the offset is non-zero and the limit is set Hibernate generates following query:
WITH query AS (SELECT
inner_query.*,
ROW_NUMBER() OVER (
ORDER BY
CURRENT_TIMESTAMP) as __hibernate_row_nr__
FROM
( select
* //replaced with * to be concise
from
single_entry singleentr0_
where
(singleentr0_.year=?
and singleentr0_.period<=?
and (singleentr0_.document_number not in (select
distinct singleentr1_.document_number
from
single_entry_modification singleentr1_
where
singleentr1_.moved=?
and singleentr1_.origin_year=?
and singleentr1_.origin_period<=?
and (singleentr1_.moved_to_period>?
or singleentr1_.moved_to_year<>?)))
or singleentr0_.document_number in (select
distinct singleentr2_.document_number
from
single_entry_modification singleentr2_
where
singleentr2_.moved=?
and singleentr2_.origin_year=?
and singleentr2_.origin_period<=?
or singleentr2_.moved_to_period<=?
and singleentr2_.moved_to_year=?))
and singleentr0_.account_number=? ) inner_query ) SELECT
* //replaced with * to be concise
FROM
query
WHERE
__hibernate_row_nr__ >= ?
AND __hibernate_row_nr__ < ?
that query returns the correct number of results.
I have 2 questions:
1. why the first query returns only 1 result?
2. why in the first query TOP(?) appears not in top level select but in subquery?
Please let me know if additional source code should be provided
Thank you!
Can you please share me code snippet to be written via JPA in order to generate the below sql query
SELECT COUNT(*) FROM Customer c
WHERE c.countryId ='Canada' AND
c.lanuguage ='ENG' AND
ROW_NUM <=10;
Because I tried in the below way. But MaxResults is not getting applied it seems as I can able to recieve the count more than 10.
Query query = em.createQuery("SELECT COUNT(c) FROM Customer c where c.countryId ='Canada' and c.lanuguage ='ENG'");
query.setMaxResults(10);
long customerCount = (Long)query.getSingleResult();
Select on count will always return a single value. If you want to have a count lower than 10, add HAVING.
SELECT COUNT(c) AS
FROM Customer c
WHERE c.countryId='Canada' and c.language='END'
HAVING COUNT(c)<=10
This is my SQL query with which I get all the duplicates, but one(the newest one):
SELECT d.C_ContactID, d.C_EmailAddress, d.C_DataSourceID, d.C_DateCreated
FROM duplicates as d
WHERE d.C_DateCreated !=(select max(d2.C_DateCreated)
FROM duplicates d2
WHERE d2.C_DataSourceId = d.C_DataSourceId)
Is it possible to optimize it somehow? Unfortunately in 300 000 records it takes +- 40minutes.
Method where the query is:
public ArrayList<Record> get() throws SQLException,
ClassNotFoundException {
Statement st = DBConnect.DBC.con.createStatement();
String sql = ("select d.C_ContactID, d.C_EmailAddress, d.C_DataSourceID,
d.C_DateCreated "
+ "from duplicates as d "
+ "where d.C_DateCreated !=(select max(d2.C_DateCreated) "
+ "from duplicates d2 where d2.C_DataSourceId = d.C_DataSourceId)");
ResultSet rs = st.executeQuery(sql);
DBConnect.DBC.con.commit();
while (rs.next()) {
int contactID = rs.getInt("C_ContactID");
String email = rs.getString("C_EmailAddress");
String dataSourceID = rs.getString("C_DataSourceID");
String dateCreated = rs.getString("C_DateCreated");
duplicate = new Record(contactID, email, dataSourceID, dateCreated);
duplicates.add(duplicate);
}
rs.close();
st.close();
return duplicates;
}
You would start by creating an index on duplicates(C_DataSourceId, C_DateCreated):
create index duplicates_DataSourceId_DateCreated on duplicates(C_DataSourceId, C_DateCreated);
If you are using a database that supports window functions, then I would rephrase this as:
SELECT d.C_ContactID, d.C_EmailAddress, d.C_DataSourceID, d.C_DateCreated
FROM (select d.*, max(C_DateCreated) over (partition by C_DataSourceId) as maxdc
from duplicates d
) d
WHERE d.C_DateCreated <> maxdc;
It is worth doing the comparison, because sometimes window functions have efficient implementations.
And, if you have the index, a slightly more efficient version of your query is:
SELECT d.C_ContactID, d.C_EmailAddress, d.C_DataSourceID, d.C_DateCreated
FROM duplicates d
WHERE EXISTS (select 1
from duplicates 2
where d2.C_DataSourceId = d.C_DataSourceId and
d2.C_DateCreated > d.C_DateCreated
);
This says to get all rows from duplicates where there is another row (with the same source) that has a bigger date created. The slight advantage is that this doesn't have to look at all the values to get the max(). It only has to find the first one. The major performance improvement will be the composite index.
Create index for the columns C_DateCreated and C_DataSourceId. This will reduce the time to execute the query. Refer this link to know how to create indices.
http://www.tutorialspoint.com/mysql/mysql-indexes.htm
In T-SQL the answer would look like this, but I don't think SQLite supports window functions (leaving the answer intact for posterity though):
You can use a window function to label each row with it's position in the group of common C_DataSourceIDs, then with a CTE select the rows that aren't in position 1 like this.
with ordered as (
select
d.C_ContactID,
d.C_EmailAddress,
d.C_DataSourceID,
d.C_DateCreated,
row_number() over (
partition by
d.C_DataSourceID
order by
d.C_DateCreated
) as rownum
from
duplicates
) select
C_ContactID,
C_EmailAddress,
C_DataSourceID,
C_DateCreated
from
ordered
where
rownum != 1;
With an index on (C_DataSourceID, C_DateCreated) this will only need a single pass over the table instead of a self join that you have in your query.