I have a SmartGWT ListGrid with 16 fields each containing Strings. The grid is attached to a data source provided via REST. I am facing issues though when the amount of data in the datasource is large.
When dealing with very small numbers there are no problems, even when dealing with 200,000 records there are still no problems. However when it comes to much larger datasets, for example one I tried earlier had 2.6 million records the grid only displays the first 850,000 (approximatley) records then refuses to page any further. Even more oddly, when my datasource has about 20 million records the grid only displays the first 20 then refuses to page at all.
Strangley I can see the data coming back in my RPC response using the isc.showConsole(); and can see that it is returning the correct data, in fact even when it returns 64 valid records it only displays the first 20. The totalRecords is properly returned as 20 million but it just will not page.
My grid is setup as follows
ListGrid grid = new ListGrid();
DataSource ds = Application.getMyDataSource();
grid.setCriteria(new Criteria("someid", 627263));
grid.setDataSource(ds);
grid.setAutoFetchData(true);
My Datasource is setup correctly as it works perfectly on smaller datasets, each field is created as a DataSourceTextField.
The server side accepts the incomming request, looks for the someid coming in, checks the _startRow, _endRow parameters finds those rows (I'm not using a database for this) and manually sets the startRow, endRow and totalRow fields.
When I inspect the data coming back the data is correct yet the grid refuses to display it correctly.
What am I missing? Is there a maximum size for list grids? From what I've seen they should handle millions of records, yet mine seems to fail completely.
EDIT
Upon further inspection it seems it is an error with the way SmartGWT handles the sizing of the scrollbar. I have no idea how to fix this at the moment.
When scrolling to the bottom of the table on what should be a 2.3 million row table, the last row displayed is 894,785. This is well short of the number of rows expected.
All data coming back from the data source is correct at this point. Interestingly clicking on the last row and scrolling with the down arrow on keyboard works fine, this to me re-enforces the point that it is a scroll bar issue.
In order for progressive loading to work, grid has to be tricked in a way to indicate there's more data.
Since we don't want to draw 1000 empty records (actual totalRows) when only 150 is fetched with data, we need to set totalRows = 160 or something to generate a close enough scroll thumb size, as well as allow scrolling below 150 records, to trigger progressive loading of next page.
Check:
Paging and total dataset length in ResultSet
http://forums.smartclient.com/showthread.php?t=516
http://forums.smartclient.com/showthread.php?t=2750
When you drag the scroll thumb too fast towards the bottom, grid might 'intelligently' skip loading intermediate rows.
ListGrid.showAllRecords might be what you need to check based on http://forums.smartclient.com/showthread.php?t=23638.
That last thread also indicates issues with browsers and large datasets.
Related
I'm transitioning some tables from Swing to FX.
I am trying to create a custom horizontal oriented TableView kind of like what is asked about here.
I utilized this method to wrap and display my column of data. I did a bind between two table scrollbars to get a "row header". Here is what the entire thing looks like:
Then I checked how well TableView handled loading a large amount of data. I created a basic multiplication table with 10 columns and 100,000 rows. After the initial load-in, the table was incredibly responsive and the vertical scrollbar movement and had no issue.
My issue came when I add more columns. I believe because of the way that TableView expects data to be in rows instead of columns that when I add 10,000 columns and 50 rows the entire TableView component was unresponsive. It also took significantly longer to load-in than the 10 columns, 100k rows.
At ~4k columns and 50 rows, the table responded well to the horizontal scroll, but the vertical was very slow to respond, which is why (apart from the inherent structure) I was lead to believe TableView prefers row data to column.
Is there a way around the unresponsiveness that preferably:
Keeps the columns as the dataset
Doesn't involve going back to JTables
Avoids pagination
I found this post, but it did not seem helpful and the OP went back to JTables in the end. On the other hand, this was 4 years ago and a slightly different case.
Please note I am new to posting so let me know if more info is needed.
I am writing a browser based application using GWT and making use of websql (yes, I know it is deprecated). I have created a custom table widget (based on FlexTable) and enabled it to scroll with the help of some CSS trickery. What I am striving to achieve (without much success) is that when the user scrolls to the start/end of the current data in the table, an event is fired and the next subset of X rows is returned from the websql DB and replaces the data currently in the table. In order for this to work, I need to keep track of the data offset in the table widget so that I can pass this to the query and use the limit and offset functions of SQL to return the required data. However, I just cannot seem to get the logic right to implement the data offset tracker within the widget. Another complication is that I need the table to be able to scroll 'into the past' (so to speak), so that it can retrieve data from before the initial start point when the table loads.
I have been at this for a number of days now and just cannot seem to get it right. So I was wondering/hoping that someone might be able to point me in the right direction (PLEASE!).
Thanks in advance
Tim
I am not sure why this is causing a problem.
int page = 0;
// when you hit the bottom
page++;
loadData(page);
// when you hit the top
if (page > 0) {
page--;
loadData(page);
}
Tim I think it is not a good idea controlling the scroll with CSS trickery.
I have done something similar soon and controlling all the logic (pagination, scroll positions,...).
What I suggest to use is a gwt's scrollPanel, a HasData widget (like a cellList) your custom AbstractCell (class which is rendered for each row of your list) and asyncDataProvider ( which gives you the onRangeChange handler for asking to your server when the range data display has changed).
You can force/fire that event when in scrollPanel.addScrollHandler detects you are arriving to the end.
If you want to see all of this in action have a look into (click on source code): http://gwt.googleusercontent.com/samples/Showcase/Showcase.html#!CwCellList
EDIT [according comment below]:
A. If you want to override the data (in the example is X+X+X...) with the new retrieved just maintain always the same range of data displayed [display.setVisibleRange(0, newPageSize);], and start from 0 when you render the new data (on your RangeChange listener).
B. If you need to have control over up and down scrolls instead of taking advantage of the used events internally on the cellList (basically onRangeChange), you can create your custom events and fire them (this option could be easier for your colleagues for understanding everything). And do not worry about controlling up and down scrolls, inside the ShowMorePagerPanel.java you can see a simple example of knowing up and down controls.
Anyway, I did not explain more detailed because I did not see you very convinced to use CellList approach (and I was using my tablet :D ).
If you change your mind, just let me know and I write for you a proper example step by step (this part could be tricky, so if you are lost it is normal ;) ).
Right now, I have an app with a ListView. In each row, there is a TextView that shows the number of times that item has been favorited, which is represented in a Favorites table.
Currently, that number is gotten by using a count(*) aggregation on the number of times that itemID appears in the table. So every time a user favorites an item, it should add a row to the table and increment the number. However, as this calculation and redrawing has to happen for every row in the ListView, it takes about 2 seconds for the changes to occur.
Are there any improvements I can make to speed this up?
I would add a "favorited_count" column to the "item" table that gets incremented whenever the item is favorited. Although this may end up duplicating information e. g. rows in the "favorited" table, it should perform a whole lot better than running count(*) on the "favorited" table, especially as it continues to grow in size.
Without any code its hard to tell, however there are a few things you might check.
Are you using View Holder pattern (suggested by google) this avoids unwanted parsing through the list item's layout every time it is drawn.
See this,
http://developer.android.com/training/improving-layouts/smooth-scrolling.html
http://www.javacodegeeks.com/2013/09/android-viewholder-pattern-example.html
Make sure your list-item layout is optimized,
http://developer.android.com/training/improving-layouts/index.html
I would like to ask question that is not directly related to any technology but it’s a logic level question. Forgive me if I am wrong.
I am trying to solve this issue in Java side.
We have a list view (may be JTable or SWT Table). We are displaying records from a table in to that list. The table may have millions of records. And the table size is keeps increasing. As the table size is huge and it keeps increasing we are keeping a cache of 3000 records and displaying the cache in the table. So at a single point only 3000 records will be there in the view. At this time, the scroll bar of the view (JTable or SWT Table) shall be set based on the cache only (in this case 3000 records).
And based on the user events like, PgUP, PgDwn, DownKey, UpKey, Mouse Scroll etc we are refreshing the cache so that the view will be refreshed.
But we need to give users an actual feel that the scroll position based on the records in the table. I think it is not possible as the scroll position is set by the OS based on the number records in the display. Can somebody can assist me with an alternate method. If I am not explained my question correctly, I will explain this in details. Please help.
Usually a scrollbar is set at the top of the list, or at the bottom, depending on if the property "AutoScroll" is set to true or false.
I believe there IS a method where you can set the scrollbar at a specific point, but last time I tried using it, it was kinda clunky.
It sounds like you're trying to make the scroll bar do two things at once, and that can be confusing. You could use a separate control to the do the paging. A row of buttons in a JToolBar looks nice, but a JComboBox will take up less room. There's some good answers about this here.
How can i keep no of rows constant in text area.
I need to create a console window for my application.
If rows exceeds predefined no of rows first rows must get disposed.
As if first written row will be destroyed first when i append anything which exceeds no of rows set.
One more thing , i need to keep vertical scroll bar. That means no of rows must not be the whatever rows are visible when text area it opened.
For example : - no of visible rows on view port are 30.
It should keep 120 rows information, which will can be seen with the help of scroll bar.
Sounds like what you want to do is create your own Document implementation. See Document.
The Message Console shows one way.