I use sqlite to store data. I am trying to get data from sqlite table view and fill array of objects in java, but Query Execution takes very long time.
I only have 32 objects with 22 fields, and sqlite with 380 rows.
But to Execute similar statement took me 17 seconds for 32 objects.
sql = "SELECT "
+ " field1,"
+ " field2,"
....
+ " field22"
+ " from Rankedview WHERE Ranking = " + Integer.toString(RankingIndex);
try (ResultSet rs = stmt.executeQuery(sql)) {
while (rs.next()) {
a[j].field1= rs.getString("field1");
..........
a[j].field22 = rs.getInt("field22");
}
}
After I updated sqlite-jdbc driver from 3.7.2 to 3.8.5 time from 17 seconds lowered to 9 seconds.
How can I improve its performance?
Edit:
view definition (ATP is a table)
CREATE VIEW Ranked AS
SELECT p1.ID,
p1.field2,
...
p1.field21,
(
SELECT count() + (
SELECT count() + 1
FROM Table AS p2
WHERE p2.field21 = p1.field21 AND
p2.id > p1.id
)
FROM ATP AS p2
WHERE p2.field21 > p1.field21
)
AS Ranking
FROM ATP AS p1
ORDER BY Ranking ASC;
EXPLAIN QUERY PLAN output:
selectid order from detail
0 0 0 SCAN TABLE ATP AS p1
0 0 0 EXECUTE CORRELATED SCALAR SUBQUERY 1
1 0 0 SCAN TABLE ATP AS p2
1 0 0 EXECUTE CORRELATED SCALAR SUBQUERY 2
2 0 0 SEARCH TABLE ATP AS p2 USING INTEGER PRIMARY KEY (rowid>?)
0 0 0 EXECUTE CORRELATED SCALAR SUBQUERY 3
3 0 0 SCAN TABLE ATP AS p2
3 0 0 EXECUTE CORRELATED SCALAR SUBQUERY 4
4 0 0 SEARCH TABLE ATP AS p2 USING INTEGER PRIMARY KEY (rowid>?)
0 0 0 EXECUTE CORRELATED SCALAR SUBQUERY 5
5 0 0 SCAN TABLE ATP AS p2
5 0 0 EXECUTE CORRELATED SCALAR SUBQUERY 6
6 0 0 SEARCH TABLE ATP AS p2 USING INTEGER PRIMARY KEY (rowid>?)
0 0 0 USE TEMP B-TREE FOR ORDER BY
To get a row with a specific rank, you should not compute the rank by hand, but use the LIMIT/OFFSET clauses:
SELECT ...
FROM ATP
ORDER BY field21, id
LIMIT 1 OFFSET x
This still requires sorting all table rows to determine which is the x-th, but is much more efficient than multiple nested table scans.
Related
I am facing an issue in Oracle Query to achieve the following use case,
Consider I have two tables :
Table 1 : product
productId - Integer - primaryKey
productName - Varchar
Table 2 : product_sequence
productId - Integer - primaryKey
sequenceId - Integer - primaryKey
orderId - Integer
orderName - Varchar
product table has 1000 entries and product_sequence table has 10K entries
Requirement :
(paginate) fetch the entries from 0 to 100 / 100 to 200 / etc., in the product table
Distinct count of productId for showing the pagination in UI (check the sample query below)
Filter by 'productName' in 'product' table and 'orderName' in 'product_sequence' table
Query (tried) :
SELECT
p.productId, p.productName, ps.orderId, ps.orderName,
COUNT(distinct p.productId) OVER () AS TOTAL
FROM (
select *
from product
OFFSET 0 ROWS FETCH NEXT 100 ROWS ONLY
) p
JOIN product_sequence ps on p.productId=ps.productId
WHERE ps.orderId IN ('12','13','14');
NOTE : the above query will work in Oracle, But the issue is
Expected:
Return 100 entries from 'product' table with mapped entries in the 'product_sequence' table
Actual :
It first LIMITS 100 entries in product and then filter the orderId so the number of entries returned is reduced from 100 to lesser number
I agree my query is not correct: It first LIMIT by 100 in 'product' table in subquery and then goes for filter in second table which reduces the count
Could some one help me with the query for this please? Anyhelp is appreciated.
If my question is not clear, Let me know, I can explain with more info.
Try to move the OFFSET and FETCH clauses to the outer query, something like this:
SELECT q.productId, q.productName, q.orderId, q.orderName,
COUNT(distinct p.productId) OVER () AS TOTAL
FROM ( SELECT * FROM product p JOIN product_sequence ps ON p.productId = ps.productId
WHERE ps.orderId IN ('12','13','14') ) q
OFFSET 0 ROWS FETCH NEXT 100 ROWS ONLY
To get 100 rows per page "after filtering" you'll need to find all the productid values first, then process the main query.
For example:
select
p.productid, p.productname, ps.orderid, ps.orderName,
count(distinct p.productid) over() as total
from product p
join product_sequence ps on p.productid = ps.productid
where ps.orderid in ('12','13','14')
and p.productid in (
select *
from (
select distinct p.productid
from product p
join product_sequence ps on p.productid = ps.productid
where p.productname like '%theremin%' -- filter #1
and ps.orderid in ('12','13','14') -- filter #2
) x
order by productid
offset 300 rows -- get the 4th page
fetch first 100 rows only -- page size set to 100 rows
)
See example at db<>fiddle.
I want to implement java application that can connect to any sql server and load any table from it. For each table I want to create histogram based on some arbitrary columns.
For example if I have this table
name profit
------------
name1 12
name2 14
name3 18
name4 13
I can create histogram with bin size 4 based on min and max value of profit column and count number of records for each bin.
result is:
profit count
---------------
12-16 3
16-20 1
My solution for this problem is retrieving all the data based on required columns and after that construct the bins and group by the records using java stream Collectors.groupingBy.
I'm not sure if my solution is optimized and for this I want some help to find the better algorithm specially when I have big number of records.(for example use some benefits of sql server or other frameworks that can be used.)
Can I use better algorithm for this issue?
edit 1:
assume my sql result is in List data
private String mySimpleHash(Object[] row, int index) {
StringBuilder hash = new StringBuilder();
for (int i = 0; i < row.length; i++)
if (i != index)
hash.append(row[i]).append(":");
return hash.toString();
}
//index is index of column for histogram
List<Object[]> histogramData = new ArrayList<>();
final Map<String, List<Object[]>> map = data.stream().collect(
Collectors.groupingBy(row -> mySimpleHash(Arrays.copyOfRange(row, index))));
for (final Map.Entry<String, List<Object[]>> entry : map.entrySet()) {
Object[] newRow = newData.get(rowNum);
double result = entry.getValue().stream()
.mapToDouble(row ->
Double.valueOf(row[index].toString())).count();
newRow[index] = result;
histogramData.add(newRow);
}
As you have considered, performing the aggregation after getting all the data out of SQL server is going to be very expensive if the number of rows in your tables increase. You can simply do the aggregation within SQL. Depending on how you are expressing your histogram bins, this is either trivial or requires some work. In your case, the requirement that the lowest bin start at min value requires a little bit of setup as opposed to binning starting from 0. See sample below. The inner query is mapping values to a bin number, the outer query is aggregating and computing the bin boundaries.
CREATE TABLE Test (
Name varchar(max) NOT NULL,
Profit int NOT NULL
)
INSERT Test(Name, Profit)
VALUES
('name1', 12),
('name2', 14),
('name3', 18),
('name4', 13)
DECLARE #minValue int = (SELECT MIN(Profit) FROM Test)
DECLARE #binSize int = 4
SELECT
(#minValue + #binSize * Bin) AS BinLow,
(#minValue + #binSize * Bin) + #binSize - 1 AS BinHigh,
COUNT(*) AS Count
FROM (
SELECT
((Profit - #minValue) / #binSize) AS Bin
FROM
Test
) AS t
GROUP BY Bin
| BinLow | BinHigh | Count |
|--------|---------|-------|
| 12 | 15 | 3 |
| 16 | 19 | 1 |
http://sqlfiddle.com/#!18/d093c/9
I am having problems using filters to search data in hbase.
First I am reading some data from one table and storing in a vector or arrayList:
for (Result r : rs) {
for (KeyValue kv : r.raw()) {
if (new String(kv.getFamily()).equals("mpnum")) {
temp = new String(kv.getValue());
x.addElement(temp);
}
}
}
Then, I want to search a different table based on the values of this vector. I used filters to do this: (I tried BinaryPrefixComparator and BinaryComparator as well)
FilterList filterList = new FilterList(FilterList.Operator.MUST_PASS_ONE);
for (int c = 0; c < x.size(); c++) {
System.out.println(x.get(c).toString());
filterList.addFilter(new SingleColumnValueFilter(Bytes.toBytes("mpnum"), null, CompareOp.EQUAL, new SubstringComparator( x.get(c).toString() )));
}
I should get 3 results back, however I only get one result back, the first entry in the database.
What doesn't make sense is that when I hardcode the value that I am looking for into my code, I will get all 3 results back.
I thought there might be some issue with converting the bytes to String and then back to bytes, but that would not explain how it was able to bring back the first result. For some reason, it is stopping at the first match and doesn't continue to find the other 2 rows that contain matching data. If I hardcode it i get the results:
x.addElement("abc123");
filterList.addFilter(new SingleColumnValueFilter(Bytes.toBytes("mpnum"), null, CompareOp.EQUAL, new SubstringComparator( x.get(0).toString() )));
Does anyone know what the problem is or what I need to do to resolve my issue? Your help is much appreciated.
Thank You
edit: Here is the contents of the tables:
TABLE1:
ROW COLUMN+CELL
0 column=gpnum:, timestamp=1481300288449, value=def123
0 column=mpnum:, timestamp=1481300273355, value=abc123
0 column=price:, timestamp=1481300255337, value=85.0
1 column=gpnum:, timestamp=1481301599999, value=def2244
1 column=mpnum:, timestamp=1481301582336, value=011511607
1 column=price:, timestamp=1481301673886, value=0.76
TABLE2
ROW COLUMN+CELL
0 column=brand:, timestamp=1481300227283, value=x
0 column=mpnum:, timestamp=1481300212289, value=abc123
0 column=price:, timestamp=1481300110950, value=50.0
1 column=mpnum:, timestamp=1481301806687, value=011511607
1 column=price:, timestamp=1481301777345, value=1.81
13 column=webtype:, timestamp=1483507543878, value=US
3 column=avail:, timestamp=1481306538360, value=avail
3 column=brand:, timestamp=1481306538360, value=brand
3 column=descr:, timestamp=1481306538360, value=description
3 column=dist:, timestamp=1481306538360, value=distributor
3 column=mpnum:, timestamp=1481306538360, value=pnum
3 column=price:, timestamp=1481306538360, value=price
3 column=url:, timestamp=1481306538360, value=url
3 column=webtype:, timestamp=1481306538360, value=webtype
4 column=avail:, timestamp=1481306538374, value=4
4 column=brand:, timestamp=1481306538374, value=x
4 column=descr:, timestamp=1481306538374, value=description
4 column=dist:, timestamp=1481306538374, value=x
4 column=mpnum:, timestamp=1482117383212, value=011511607
4 column=price:, timestamp=1481306538374, value=34.51
4 column=url:, timestamp=1481306538374, value=x
4 column=webtype:, timestamp=1481306538374, value=US
5 column=avail:, timestamp=1481306538378, value=
5 column=brand:, timestamp=1481306538378, value=name
5 column=descr:, timestamp=1481306538378, value=x
5 column=dist:, timestamp=1481306538378, value=x
5 column=mpnum:, timestamp=1482117392043, value=011511607
5 column=price:, timestamp=1481306538378, value=321.412
5 column=url:, timestamp=1481306538378, value=x.com
THIRD TABLE (to store result matches)
0 column=brand:, timestamp=1481301813849, value=name
0 column=cprice:, timestamp=1481301813849, value=1.81
0 column=gpnum:, timestamp=1481301813849, value=def2244
0 column=gprice:, timestamp=1481301813849, value=0.76
0 column=mpnum:, timestamp=1481301813849, value=011511607
**should be three matches those that are in bold above but only brings back one match
If anyone is willing to help for a fee, send me an email at tt224416#gmail.com
I have a table called friendgraph(friend_id, friend_follower_id) and I want to calculate 6-degrees of separation for a given friend and a given degree.
The table looks like this:
friend_id, friend_follower_id
0,1
0,9
1,47
1,12
2,41
2,66
2,75
3,65
3,21
3,4
3,94
4,64
4,32
How do I go and built a query where given friend_id_a, and order_k, find the users who are k degree apart from friend_id_a?
This is how my initial query file looks like:
create or replace function degree6
(friend_id_a in integer, order_k in integer)
return sys_refcursor as crs sys_refcursor;
I am looking for any kind of help or resources that will get me started and eventually arrive at the output.
UPDATE:
The output would be a list of other friends k degrees apart from friend_id_a.
Define order-k follower B of A such that B is not A, and:
1. if k = 0, then A is the only order-0 follower of A.
2. if k = 1, then followers of A are order-1 followers of A.
3. if k > 1; then for i = 0 to k-1, B is not order-i follower of A; and B is a follower
of a order-(k-1) follower of A
Thanks.
You can build a hierarchical query and filter by level and friend_id. For example to get all friends of user 0 at level 3:
SELECT friend_id, friend_follower_id, level
FROM friends
WHERE LEVEL = 3
CONNECT BY PRIOR friend_follower_id = friend_id
START WITH friend_id = 0
On Oracle 11gR2 or later we can use the recursive subquery-factoring syntax to do this.
with friends (id, follower, lvl) as
( select friend_id, friend_follower_id, 1
from friendgraph
where friend_id = 0
union all
select fg.friend_id, fg.friend_follower_id, f.lvl + 1
from friendgraph fg
join
friends f
on (fg.friend_id = f.follower)
where f.lvl + 1 <= 3
)
select *
from friends
/
Here's one way of implementing this in a function with a Ref Cursor:
create or replace function degree6
(friend_id_a in integer
, order_k in integer)
return sys_refcursor
is
return_value sys_refcursor;
begin
open return_value for
with friends (id, follower, lvl) as
( select friend_id, friend_follower_id, 1
from friendgraph
where friend_id = friend_id_a
union all
select fg.friend_id, fg.friend_follower_id, f.lvl + 1
from friendgraph fg
join
friends f
on (fg.friend_id = f.follower)
where f.lvl + 1 <= order_k
)
select *
from friends;
return return_value;
end degree6;
/
Using it in SQL*Plus:
SQL> var rc refcursor
SQL> exec :rc := degree6(0,3)
PL/SQL procedure successfully completed.
SQL> print rc
ID FOLLOWER LVL
---------- ---------- ----------
0 1 1
0 9 1
1 12 2
1 47 2
SQL>
I am trying to create a nested SELECT SQL statment. I store all value and id and want to select rows that satisfy multiple values. How can I generate a SELECT statement using Java? For example,
ID VALUE
1 RED
2 BIG
1 SMALL
1 SMOOTH
2 TALL
.....
To select an item that is both red and small the statement would be:
SELECT *
FROM table
WHERE table.value = RED AND
id IN (SELECT *
FROM table
WHERE table.value = SMALL AND id IN (...))
This type of problem is called Relational Division
SELECT ID
FROM tableName
WHERE VALUE IN ('SMALL','SMOOTH')
GROUP BY ID
HAVING COUNT(*) = 2
SQLFiddle Demo
the query above will result 1 since the ID contains both records.
If no unique constraint was enforce on value for every ID, DISTINCT is required.
SELECT ID
FROM tableName
WHERE VALUE IN ('SMALL','SMOOTH')
GROUP BY ID
HAVING COUNT(DISTINCT VALUE) = 2
SQLFiddle Demo
SQLFiddle Demo (with duplicate)
OTHER(s)
SQL of Relational Division
select ID
from MyTable
where VALUE in ('RED', 'SMALL')
group by ID
having count(distinct VALUE) = 2
SQL Fiddle Example
Results:
| ID |
------
| 1 |
Here is a general way to approach this:
select id
from t
group by id
having max(case when value = 'Red' then 1 else 0 end) = 1 and
max(case when value = 'Small' then 1 else 0 end) = 1
In other words, membership in the set becomes a clause in the having statement. These can be both inclusion and exclusion (use = 0 instead of = 1) and optional (use or instead of and).