I have a table in Oracle: UserDetail[id, name, country]. The problem statement is to get all 'UserDetail' entities whose 'name' is from a list of given input list using Hibernate. The most obvious solution is using 'Restrictions.in' from hibernate criteria api:
//Session Construction code
List<String> usernames = getUserNames();
Criteria criteria = session.createCriteria(UserDetail.class);
criteria.add(Restrictions.in("name", usernames)); //usernames -> List of usernames
List<UserDetail> users = criteria.list();
The question is will there be any performance issues if the size of list(usernames) is like 10k and the number of users present in the database is roughly around 10 million. Would like to know what the performance issue will be and what can be an alternative way to get the data set with this kind of filter.
Thanks in advance
If this list of user names is determined through some kind of query, I would recommend you embed that query as subquery instead as that will usually perform better. If the list is provided by a user, there is not much you can do other than what you already have.
Related
Should we query the table with more filtering by adding multiple conditions/ where clauses to SQL query and get the specific data
Or pull all the data and do filtering in our java class.
Looking for efficient way of coding practices
Example :
A table with multiple columns Id, Name, Place.
I need to pull the list of ids with Place should be form placesList and Name should match namesList.
1)
(SELECT id
FROM Person p
WHERE p.name IN (<name_list>)
AND p.place IN (<place_list>)
order
by p.id asc)
public List<Long> getIds(#BindIn("name_list") List<String> name_list, #BindIn("place_list") List<String> place_list);
or
2)
(SELECT id
FROM Person p)
public List getIds();
apply java8 filters to the result
Note: Above example i took name place for easy understanding. In Real time, data is huge and have multiple fields and rows in the table. The list used to filter is also large.
Best approach is query with required filters on database and which will reduce amount of data fetch you from applicaion and db back and forth and also it will reduce time on I/O operations(since it involves some latency to transfer large amount of data over network).
also reduces overhead of memory need to process large amount of data on application side.
Also when you are running query and filtering on multiple fields you can add indexes(if necessary) on fields it will improve query fetch time.
Hope it answers
You always want to perform things in the database if possible. You want to avoid transferring data from the database to your application, using up memory, just to remove it there.
Databases are very efficient at doing those things, so you'll want to use them to their full extent.
Query the database directly instead of downloading data in Java application. It will reduce the latency from the database to your java application.
But be very careful when using user inputs in the filter. Make sure that you have sanitized the user input before using them in a query to the database to avoid SQL injection.
If you are worried about security more than performance then filter the data in Java App (if the data is not massive in size).
But I strongly recommend filtering the data on the database itself by ensuring necessary safeguards.
I have a Employee object for example, Employee contains several fields (300+) like name, department, salary, age, account, etc.
Entire Employee table data cached into java List object , which contains 2+ million records.
Requirement
user can search on any filed presents in Employee object like employee name like Sehwag and age > 30 or salary > 100000, Based on user search we have to show filtered list of Employee list.
due to performance issues we are not querying DB, we want to apply the user search criteria on cached java List object earlier
is there way api / frameworks / any other solution where we can query on java objects?
below approach I am trying but I am feeling not a good approach
Iterating the Employee list and applying condition user search criteria on employee object, to know the user selected search criteria among 300 fields is challenging, written a lot enum mapping logic and some additional logic for every filed to make it work.
with current requirement it may works but thinking to use api or framework or better way to solve the requirement!
thanks in advance for your help.
First of all. If you either don't want or don't have chance to change the existing solution, take a look at querydsl.
There is a querydsl-collection module which fits exactly with your need. See more at http://www.querydsl.com/ and http://www.querydsl.com/static/querydsl/latest/reference/html/ch02s08.html
However, if you have a chance to review/rebuild the solution, you should consider something more appropriate for large volume querying. I suggest you exploring more about nonsql databases (mongodb) or indexing tools such as lucene or elasticsearch which adds a RESTFul layer on top of lucene.
I hope it helps.
tried CQEngine's SQL based queries https://github.com/npgall/cqengine suits to my requirement,
below are some useful links
https://dzone.com/articles/getting-started-cqengine-linq
https://mvnrepository.com/artifact/com.googlecode.cqengine/cqengine
I am trying delete records in bulk from database table Student.
Everything is running fine but my question is :
Is there any limitation when my list (refer to query: studentIdList) size is greater than 1000000 in given piece of code? Do I need to do anything extra in such situation?
String hql = "delete from Student where id in (:studentIdList)";
session.createQuery(hql).setParameterList("studentIdList",studentIdList).executeUpdate();
session.flush();
There are few things to be considered.
1.) how cache will behave if configured.
2.) For 1000000 records, definitely load test needs to be done. Are there any changes of OOM error.
You can try with above HQL and Hibernate Batch and then measure and come up with the statistics. Blindly telling the stats is impossible.
Also instead of using IN operator, how about the performance of using == in loop.
WHERE id = 1;
Is transformed to a simple equality filter.
WHERE id IN (1);
Is transformed into an array match of:
WHERE id = ANY(ARRAY[1]);
I will suggest you to please try, it will be an interesting exercise for you.
After reading more and experimenting, Batch will internally make a final query with IN operator only.SO using batch will add extra process in between, my thought using HQL with IN for delete operation will be good to go.
I'm working on a feature which allow some users to define their own SQL queries and run them on the database.
Basically a query could look like this:
1. SELECT first_name, last_name FROM user;
2. SELECT first_name, last_name, id, address, email FROM user.
As you can see there may be a different number of columns in the result table.
Is there a way to handle this in Hibernate?
For instance, the basic usage displayed below does not help me in any way because I cannot be sure that each result row has at least 2 columns.
Query query = session.getSession().createSQLQuery(queryStr);
ScrollableResults results = query.scroll(ScrollMode.FORWARD_ONLY);
while (results.next()) {
data.put("firstName", String.valueOf(results.get(0)));
data.put("lastName", String.valueOf(results.get(1)));
}
Furthermore, I don't think I can use the select new map because the users have to run native MySQL queries.
Is there any solution to this?
Thanks in advance for your time and suggestions!
First of all, I do not think that it is a good idea to expose an interface that allows SQL input to users.
Anyway, in your case you could program the SQL result set extraction yourself. There is no need for object relational mapping, if you do not map into objects. You could then go and just check for the existence of the columns and map them right away into your data structure. Nice way to learn plain JDBC. Just for the case you use Hibernate for other issues in your application, you can even mix the approaches.
I have implemented usage of ScrollableResults for big DB table, everything worked perfectly until I wanted to do the same for another table using joins.
The entity where I have the problem has some one-to-many associations so I have to use DISTINCT not to get duplicates. Everything works well when I am obtaining results of the query using list(). But when I use scroll(), DISTINCT seems to be ignored completely - I just get many duplicates.
Query query = gameSession.createQuery("SELECT DISTINCT c FROM City c JOIN FETCH c.inhabitans i");
This works well, list has no duplicates:
List<City> list = query.list();
This is does not work (giving many duplicates like there would be no DISTINCT used):
ScrollableResults sr = query.scroll(ScrollMode.FORWARD_ONLY);
Everything is the same when I use Criteria instead of Query. I have found out only 3 things about this particular problem:
Few questions like mine without answer,
A bug describing a case that could be absolutely the same like mine, but that should have been fixed long time ago,
Little comment in one of the SO answers telling that "DISTINCT_ROOT_ENTITY does not interact very well when scroll() is used".
This makes ScrollableResults useless for me, but I still need it because of the huge memory save. Do you know how to achieve scrolling results with DISTINCT used? Or any workaround?
Hibernate version: 4.2.4; JDK 7; DB: MSSQL
Add an "order by" clause to your query with the Id of the root entity.