I am trying to rewrite this query to improve performance given that it takes more than 10 mins to execute. I believe the issue is mainly due to the large Inbox table. As union isn't an option in HQL, what can I do to improve this performance? Thanks in advance.
select distinct p from PrReq p, Inbox i where i.documentNo in
(select distinct kdn.kdnId from PrKdn kdn where kdn.prReqId = p ) or
i.documentNo in (select distinct grn.grnId from PrGrn grn where
grn.prReqId = p) and i.inboxStatus = 0 and i.moduleStatus =
:currentStatus and p.recvDept.code = :dept and p.organization = :org
order by p.billToDept, p.prId ASC, p.createDate desc
My server would retrieve the latest ID from the database, now it is stuck and keeps returning the id 99999, even though the latest id is now 100040
My code is:
String insertTable = "SELECT * FROM dutyofcare ORDER BY Id DESC LIMIT 1";
ps = conn.prepareStatement(insertTable);
rs = ps.executeQuery();
String ResultS = "";
if (rs.next()) {
ResultS += rs.getString("Id");
}
The issue is that the ORDER BY in your query is doing a lexical (character-by-character) sort where 9 always comes after 1, and not numeric sort which handles the digit positions. This is because of the column type of ID. What you need is to ensure ID is a number before the sort is done.
Either change your ID to a numeric column type and run below query:
SELECT MAX(ID) from dutyofcare;
Or if you want to retain your column type (less efficient than above option):
select MAX(cast(ID AS UNSIGNED)) from dutyofcare;
Or if you want to retain your column type AND just fix your existing query (least efficient of all the options)
select * from dutyofcare order by CAST(ID AS UNSIGNED) desc limit 1;
All these methods basically treat the ID as number and choose the biggest value.
I am using queryDSL JPA.
I am trying to load data in chunks, so I am using .offset() and .limit()
I am querying 'single_entry' table, that contains 10000 records.
I use the following function to query data
private Collection<SingleEntry> getSingleEntries(FiscalPeriod fp, Predicate filter, Integer offset, Integer limit) {
BooleanExpression initialFilter = getSingleEntriesFilter(fp);
return query.selectFrom(QSingleEntry.singleEntry).where(initialFilter.and(filter)).offset(offset).limit(limit).fetch();
}
private BooleanExpression getSingleEntriesFilter(FiscalPeriod fp) {
int period = fp.getPeriod();
int year = fp.getYear();
QSingleEntryModification sem = QSingleEntryModification.singleEntryModification;
QSingleEntry se = QSingleEntry.singleEntry;
BooleanExpression fiscalPeriodMatches = se.year.eq(year).and(se.period.loe(period));
//excludes single entries that moved from this period
BooleanExpression excludeMoved = se.documentNumber.notIn(query.selectDistinct(sem.documentNumber).from(sem)
.where(sem.moved.eq(true)
.and(sem.originYear.eq(year)
.and(sem.originPeriod.loe(period)).and(sem.movedToPeriod.gt(period).or(sem.movedToYear.ne(year))))));
BooleanExpression movedToPeriodOrNonMoved = se.documentNumber.in(query.selectDistinct(sem.documentNumber).from(sem)
.where(sem.moved.eq(false).and(sem.originYear.eq(year).and(sem.originPeriod.loe(period))).or(sem.movedToPeriod.loe(period).and(sem.movedToYear.eq(year)))));
return fiscalPeriodMatches.and(excludeMoved).or(movedToPeriodOrNonMoved);
}
Hibernate produces the following query when offset equals 0 and limit are set:
select
* //replaced with * to be concise
from
single_entry singleentr0_
where
(
singleentr0_.year=?
and singleentr0_.period<=?
and (
singleentr0_.document_number not in (
select
distinct TOP(?) singleentr1_.document_number
from
single_entry_modification singleentr1_
where
singleentr1_.moved=?
and singleentr1_.origin_year=?
and singleentr1_.origin_period<=?
and (
singleentr1_.moved_to_period>?
or singleentr1_.moved_to_year<>?
)
)
)
or singleentr0_.document_number in (
select
distinct singleentr2_.document_number
from
single_entry_modification singleentr2_
where
singleentr2_.moved=?
and singleentr2_.origin_year=?
and singleentr2_.origin_period<=?
or singleentr2_.moved_to_period<=?
and singleentr2_.moved_to_year=?
)
)
and singleentr0_.account_number=?
This query return one result, however, if I run this query in a database manager, it returns a correct number of results.
If the offset is non-zero and the limit is set Hibernate generates following query:
WITH query AS (SELECT
inner_query.*,
ROW_NUMBER() OVER (
ORDER BY
CURRENT_TIMESTAMP) as __hibernate_row_nr__
FROM
( select
* //replaced with * to be concise
from
single_entry singleentr0_
where
(singleentr0_.year=?
and singleentr0_.period<=?
and (singleentr0_.document_number not in (select
distinct singleentr1_.document_number
from
single_entry_modification singleentr1_
where
singleentr1_.moved=?
and singleentr1_.origin_year=?
and singleentr1_.origin_period<=?
and (singleentr1_.moved_to_period>?
or singleentr1_.moved_to_year<>?)))
or singleentr0_.document_number in (select
distinct singleentr2_.document_number
from
single_entry_modification singleentr2_
where
singleentr2_.moved=?
and singleentr2_.origin_year=?
and singleentr2_.origin_period<=?
or singleentr2_.moved_to_period<=?
and singleentr2_.moved_to_year=?))
and singleentr0_.account_number=? ) inner_query ) SELECT
* //replaced with * to be concise
FROM
query
WHERE
__hibernate_row_nr__ >= ?
AND __hibernate_row_nr__ < ?
that query returns the correct number of results.
I have 2 questions:
1. why the first query returns only 1 result?
2. why in the first query TOP(?) appears not in top level select but in subquery?
Please let me know if additional source code should be provided
Thank you!
Can you please share me code snippet to be written via JPA in order to generate the below sql query
SELECT COUNT(*) FROM Customer c
WHERE c.countryId ='Canada' AND
c.lanuguage ='ENG' AND
ROW_NUM <=10;
Because I tried in the below way. But MaxResults is not getting applied it seems as I can able to recieve the count more than 10.
Query query = em.createQuery("SELECT COUNT(c) FROM Customer c where c.countryId ='Canada' and c.lanuguage ='ENG'");
query.setMaxResults(10);
long customerCount = (Long)query.getSingleResult();
Select on count will always return a single value. If you want to have a count lower than 10, add HAVING.
SELECT COUNT(c) AS
FROM Customer c
WHERE c.countryId='Canada' and c.language='END'
HAVING COUNT(c)<=10
I am getting ORA-01795 error in my Java code while executing more than 1000 records in IN clause.
I am thinking to break it in the batch of 1000 entries using multiple IN clause separated by OR clause like below:
select * from table_name
where
column_name in (V1,V2,V3,...V1000)
or
column_name in (V1001,V1002,V1003,...V2000)
I have a string id's like -18435,16690,1719,1082,1026,100759... which gets generated dynamically based on user selection. How to write a logic for condition like 1-1000 records ,1001 to 2000 records etc in Java. Can anyone help me here?
There are three potential ways around this limit:
1) As you have already mentioned: split up the statement in batches of 1000
2) Create a derived table using the values and then join them:
with id_list (id) as (
select 'V1' from dual union all
select 'V2' from dual union all
select 'V3' from dual
)
select *
from the_table
where column_name in (select id from id_list);
alternatively you could also join those values - might even be faster:
with id_list (id) as (
select 'V1' from dual union all
select 'V2' from dual union all
select 'V3' from dual
)
select t.*
from the_table t
join id_list l on t.column_name = l.id;
This still generates a really, really huge statement, but doesn't have the limit of 1000 ids. I'm not sure how fast Oracle will parse this though.
3) Insert the values into a (global) temporary table and then use an IN clause (or a JOIN). This is probably going to be the fastest solution.
With so many values I'd avoid both in and or, and the hard-parse penalty of embedded values, in the query if at all possible. You can pass an SQL collection of values and use the table() collection expression as a table you can join your real table to.
This uses a hard-coded array of integers as an example, but you can populate that array from your user input instead. I'm using the built-in collection type definitions, like sys.odcinumberlist, which us a varray of numbers and is limited to 32k values, but you can define your own table type if you prefer or might need to handle more than that.
int[] ids = { -18435,16690,1719,1082,1026,100759 };
ArrayDescriptor aDesc = ArrayDescriptor.createDescriptor("SYS.ODCINUMBERLIST", conn );
oracle.sql.ARRAY ora_ids = new oracle.sql.ARRAY(aDesc, conn, ids);
sql = "select t.* "
+ "from table(?) a "
+ "left join table_name t "
+ "on t.column_name = a.column_value "
+ "order by id";
pStmt = (OraclePreparedStatement) conn.prepareStatement(sql);
pStmt.setArray(1, ora_ids);
rSet = (OracleResultSet) pStmt.executeQuery();
...
Your array can have as many values as you like (well, as many as the collection type you use and your JVM's memory can handle) and isn't subject to the in list's 1000-member limit.
Essentially table(?) ends up looking like a table containing all your values, and this is going to be easier and faster than populating a real or temporary table with all the values and joining to that.
Of course, don't really use t.*, list the columns you need; I'm assuming you used * to simolify the question...
(Here is a more complete example, but for a slightly different scenario.)
I very recently hit this wall myself:
Oracle has an architectural limit of a maximum number of 1000 terms inside an IN()
There are two workarounds:
Refactor the query to become a join
Leave the query as it is, but call it multiple times in a loop, each call using less than 1000 terms
Option 1 depends on the situation. If your list of values comes from a query, you can refactor to a join
Option 2 is also easy, but less performant:
List<String> terms;
for (int i = 0; i <= terms.size() / 1000; i++) {
List<String> next1000 = terms.subList(i * 1000, Math.min((i + 1) * 1000, terms.size());
// build and execute query using next1000 instead of terms
}
In such situations, when I have ids in a List in Java, I use a utility class like this to split the list to partitions and generate the statement from those partitions:
public class ListUtils {
public static <T> List<List<T>> partition(List<T> orig, int size) {
if (orig == null) {
throw new NullPointerException("The list to partition must not be null");
}
if (size < 1) {
throw new IllegalArgumentException("The target partition size must be 1 or greater");
}
int origSize = orig.size();
List<List<T>> result = new ArrayList<>(origSize / size + 1);
for (int i = 0; i < origSize; i += size) {
result.add(orig.subList(i, Math.min(i + size, origSize)));
}
return result;
}
}
Let's say your ids are in a list called ids, you could get sublists of size at most 1000 with:
ListUtils.partition(ids, 1000)
Then you could iterate over the results to construct the final query string.