I have a user table & a child table Post containing user's posts. A user has a large number of posts just like twitter posts.
I need to fetch user posts on the UI, load more posts as the user scrolls down the page just like facebook/twitter do. I am using hibernate as the ORM framework & MySql as db. I looked into pagination found two primary ways of achieving that
1. setFirstResult();
setMaxResults();
2. ScrollableResults
I have two questions
A. Which way of pagination would be more appropriate & efficient to achieve this? I've read that ScrollableResults is more efficient than setFirstResult but it keeps the connection open for the entire pagination process.
B. As the user's profile page is loaded ajax call is fired to display the user's posts everytime, it's like a certain default content in the page. So do I need to implement a second level cache in order to avoid db hits every time the page loads?
If the dataset is large then choosing ScrollableResults will be
better.
If the users will be accessing your application with low bandwidth
and mostly will be retrieving only first or specific page say from page1 to page2
then go with setFirstResult() and setMaxResults().
Related
From a Java application, if I need to fetch 100 000 records from any RDBMS. What are the things I should consider? will it be fetched by a simple select statement?
What are the things I should consider?
The most obvious thing is that it could take a long time to transfer 100,000 "huge" records over a JDBC connection.
You might want to look at alternatives ... like a database specific data extraction tool of some kind.
will it be fetched by a simple select statement?
If you are willing to wait long enough for the transfer to compete, yes.
Suppose that application has to fetch all the records and display in UI in a J2EE application and the application is using Spring MVC with Hibernate as the ORM layer.
More things to consider:
Attempting to display 100,000 records to a user in a single page is a bit crazy. No user is going to want to scroll through 100,000 records.
If you are doing this kind of thing via an ORM, you are liable to us an inordinate amount of server-side memory.
My advice: don't fetch all 100,000 records. Instead, fetch the first N, and implement a scheme that allows the user to page through the records.
When I make an app, I always wonder when is the best moment to load data from a remote database.
If the app ask for a login screen, then:
Is it better to launch every data from remote DB at his time, and then use them in the app? This way, the app is much faster after the login screen (no more queries needed to retrive some data)
Is it better to load data only when we need to use them (for example, display some data from DB into the app). This way, the login is much faster, but the app can be slower than the first case.
What do you guys think?
It depends on the scenario, but I would go with option 2: First, the user is granted the access and you retrieve the data from your backend when needed.
Imagine a very simple scenario where the user logs in to see a list of products he can work with (add and remove). The solution would consist of 2 screens:
A login screen to manage user access
product list screen to work with products
For me, in a good design each screen has its own resposibility, and each screen should query only the data that it is going to manage.
The login screen responsibility would be only to perform the login and then navigate to the product screen.
The product screen responsibility would be retrieve the products, show them and store them when the user has ended the editings.
If your login screen queries the product data to pass it to the product screen, you are coupling the login screen and the product data.
Anyway, if you have a set of static data that can be used by several screens (for example product categories), you can query them the first time you need them and store them in a cache for further accesses.
Another scenario could be if there can be conectivity problems. In this case the best solution could be to download a set of data that the user can work with, edit them and them upload them to the backend (taking into account the possible concurrency issues).
If you want to logIn your user you have to call the remote DB to authenticate the user. I do not see where you are able to do this without the remote db.
Otherwise you are correct that you should only query the DB if you want to show the data you get.
So far I've seen examples that use the following logic:
Create a table / grid object
Set its data source (Collection such as array list/ set)
The table shows the entries on the client side!
Problem is, we have millions of rows to display, (on a side note I tried to load the container with all the entries, it took tons of time, and the client side performance were lacking)
So that raises the question:
How do you show huge amount of data on the zk tables \ grids? Wishful
thinking points me to think that instead of an array list data source
i could set a DB connection or something instead, and that will manage
the results on demand with paging.
Any ideas?
Why loading data when you are not displaying all the rows at a time.
Retieve only those data that should be displayed and load the other data on demand not while the page is initially loading.
if you try to fetch 1 million rows and try to bind it to a control, it will hugely effect your application performance and increases the time for your page to load.
So, my adivce should be fetch only those rows that needs to be displayed. if the request comes from the user for the next set of pages then load that data and bind.
you can make ajax calls to avoid every time entire page refershing
hope this helps..
ZK give BigListbox to show huge record
There have been several Play Framework pagination questions but all have been using JPA or ebean. I need to paginate data I am getting returned from a web service. Is there a way to do this with the Play Pagination Module or am I stuck with jQuery? I am also new to play and Java coming from asp.net MVC. The web service is returning a List of whatever model I am querying.
You should not paginate results from web service in the Play's controller as it would be not optimal, consider 3 scenarios (in that order)
Let's say that you want to display 10 items at once, but your generator returns 100.000 for sample query (which means 10.000 of pages)
Pagination should be done by data generator (web service in this case), so you should sent a query containg data: what are you looking for, how big page you want to get and which page you need ie: ?q=pagination&size=10&page=123. This will respond with List of 10 items on page 123. If you have possibility to change the web service to add pagination it's best choice.
If web service doesn't offer pagination get all results at once and use jQuery to paginate it. So if you'll get set of 100.000 items put all of them to the client and make sure that every page change won't cause querying the web service. Poor option, but still better than next one.
You can iterate returned results to split it easily to the pages, count total amount, etc, etc. If you'll use controller for paginating it, at every page change it will be fetching 100.000 items and then splitting to display just 10 of them. Drama :)
So if you can't use first option and don't want to use jQuery make some caching of the results on your server (ie, by storing results in the database as a separate rows) - in such case you will be able to use Ebean for local searching and paging.
I am using CachedRowSet in the java application to pull data from database. Suppose, if a query return 10,000 rows, my app will hang till i got all the rows.
Instead of that, i need to show the complete result only on scrolling. Please help me to solve this problem.
You may choose to implement your own lazy loading List which will load values page by page as accessed. There are many tutorials on this on internet check this one : http://blog.projectnibble.org/2009/07/22/java-generic-paged-lazy-list-with-jsfjpa-example-implementation/