I have a database query returning a large number of rows. JDBC doesn't tell you how many rows come back and there are too many rows to count them before I display it in a JTable.
So the model I'm working with is an Iterable<R>, but unfortunately, TableRowModel has to know the row count up-front.
I can imagine a few strategies which might work:
Load all rows, but do it in the background and add them to the model in batches.
Somehow detect requests to render the lower rows (via the model I guess) and dynamically load more rows if this happens.
Have a button on the screen to explicitly load more rows.
Some kind of non-JTable component which supports rendering rows without knowing the row count in advance (ideal, but I can't find one.)
I am wondering if there is a "normal" way to do this, because it isn't often that I see this sort of thing in a UI. The few JDBC tools I have checked out (I figured these would be the best bet) seem to either eagerly load the results or page them (as a user, I really dislike paging in GUI apps, so I would like to avoid that.)
Operations like "Find" over the table should ideally work in an "understandable" way though... so it seems like this is not going to be too easy.
Use a SwingWorker in your implementation of AbstractTableModel. Let the worker partition the query so that a reasonable number of records are available promptly, while the rest are read in the background. Use a PropertyChangeListener to show progress and implement a cancel button.
You have 2 possibilities:
Write query, which returns the number of rows, and then implement
table model, which tries to load rows from the database, if table
want to show the rows which are not in the model. In this case you need the third query,
which can index based load the rows (these queries are possible in
MS-SQL and Oracle). For example: you loads the first 100 rows, and
the number of results. If table model is asked for rows from
position 100 to 199 you makes new query, which loads these rows
interval (if it's possible for your database) or you tries to
iterate over your result set, and load the next data batch. But if
user presses the "End" key you definitly needs the index based
loading (or you need to iterate over the complete result set).
You write model, which shows, the first batch (for example 100
rows). If user, scrolls to view the last row (model.getValueAt(row,
col) -> col == 99) you simply asks the result set for next batch and
add the new rows to the model.
First variant is better for the user (because the user can see how large is the table and can directly scroll to last row), but the second is easier to implement because it does not need index based loading of rows.
Related
I'm writing an application that allows each user to label English words in three categories (some lexical exercise).
The main DB table, Word, contains ~4K different rows of words.
The Label table contains 3 labels.
--> The Word-Label table (that contains 3 columns: word_id, label_id, user_id) will add 4K rows per user (let's assume all the words starts with some pre-defined label when user register to the system).
The problem is that the table will grow very fast. 1:4000 (user/row) is bad in my opinion.
What can you suggest here to eliminate such a huge table? I've read that table-per-user is also considered bad practice.
In addition, I'm using Spring & Hibernate and the 4K insertions after the user get registered for the first time is pretty tough and takes time.
I can consider some NoSQL solution or another tool than Hibernate, but I'm consisting to use Spring & Java - so suggest something properly.
Will be glad for your help here!
There is no issue with data size. You may have an issue with Hibernate, but that is another issue.
If you end up with thousands of users, you'll have a few tens of millions of rows. That is not a large number of rows. If you want to insert default labels for a new user, then the code would look something like this:
insert into userLabels (userId, wordId, label)
select :userId, w.wordId, <default label>
from words w;
I would be surprised if this took more than a second or two.
If you knew that you would be having millions of users, then size might be more of an issue. The best solution would require better understanding of the application. The solution might vary from partitioning the tables, using arrays, or coming up with a different structure for representing your data.
You probably want various indexes on your tables to speed performance, but that depends on the queries you want to run. You might consider using a native interface to the database. Your use-case doesn't seem particularly complicated, so I don't know what advantage Hibernate or similar layers gets you.
First approach, you will just add new row to word-label for user after action. So, not every user will probably have 4k rows in that table. Now, when your database - query and stuff around that functionality will be a problem (bottleneck) then try to fix the issue and improve performance.
There are many performance tricks in sql databases you can use. For example, you wrote about table per user. That's not quite the best solution, next example, in mysql, u can create table patitions and it will be handled as one table but with performance improvement.
Second approach, for this type of data, of cource some NoSQL like MongoDB would perform great.
you could encode the user responsse-map into a 4000 entry bit-array, or string if you don't need the relational capabilities of the database
then it would be one record per user.
create table user_words (userid int, wiorddata text);
insert into user_words values (1,'YNYYNmmmYY'/* ... */ );
you application would need to have the list of words and kniow which wird each character refers to.
I am having 50000 entries in MySQL DB, which need to be fetches and iterated in java. Is it required to do pagination ?
This table could contain 20 columns which are varchar(100).
If its not possible, Is pagination required if I am fetching only 1 column from each row for these 50000 entries ?
This very much depends on your use case. If you want to display it to an end user you will very likely need pagination as displaying this much data in one go may make your UI sluggish (apart from not being very user-friendly). If you just need to do calculations in a background task, you can probably live without pagination.
I'm creating a hypermedia driven RESTful API that will be used to query transactional data. The intention is that the results will be paginated.
Each API call will query an indexed database table. Since I don't want to keep the results server side due to memory considerations, I was thinking to retrieve the data based on rownum, dependent upon which page is requested. E.G. on page one, WHERE rownum <= 10, on page two, WHERE rownum BETWEEN 11 AND 20 etc.
However, the database in question is replicated from a production system and could potentially add records into an area of the result set already requested. E.G. page one is requested -> 10 rows are returned -> a transaction is inserted at row 5. Now page two will include a record already displayed on page one, as the results are essentially pushed up by a rownum.
What would be a good way of achieving my objective of creating a hypermedia driven RESTful API that provides paginated transactional data from a database, without holding on to the result sets for the duration of the session?
This is a pretty common problem and there are actually not many approaches.
I can think of only three, actually:
You don't care and the result will change. This is the behaviour of stackoverflow: if you're on page 2 of the questions page and someone posts a new question, when clicking on page 3 you may get one or more of the questions that were already listed on page 2, because the index has shifted.
If you don't want to keep in memory the actual data, you're in for a lot of trouble. You could store the handler for the result set, instead of the results themselves, and loop over it fetching the number of rows that you actually need. E.g. you run the select, fetch 10 rows and store the handler of the resultset. Together with the rows, you return to the client a unique ID of the query. The problem will be when you have a range specified, because you can't really "rewind" a database cursor, and that would mean caching the results, which you may want to do anyway. But if you do it like that, sooner or later you're going to have all of the results in memory anyway.
You could still use some memory, but keep only some unique identifier of the rows, associated with a unique identifier of the query, as above. This could work, but only if the rows may be added, and not deleted or updated (if they're updated, they may not match the query any more).
Personally, I'd go with option 1.
I need a JTable with insert behavior that is different the one TableRowSorter/DefaultRowSorter provide.
I need a table such that when user inserts a row, the newly inserted row should go below the one where the cursor was, and cursor placed at the newly inserted row.
Let me illustrate it on an example:
Initial state - table has 4 rows:
0. Row 1
1. Row 2 <-- cursor here
2. Row 3
3. Row 4
User's cursor is at the second row, and user triggers insertion. The newly inserted row appears below, and cursor moves there as well:
0. Row 1
1. Row 2
2. New row <-- cursor here
3. Row 3
4. Row 4
I understand pretty well that I have two approaches to solve this problem.
First approach is to change my model, so insertion is done at right place in the model. I do not like this solution as it means model indexes will change in all rows below the inserted one. This is the easiest way to accomplish what I need, but, as i said, at the cost of potentially big overhead caused by model indexes change...
Second approach is to implement a RowSorter that will maintain own videToModel[] and modelToView[] arrays and maintain right order when rows are inserted. Naturally, I started by extending TableRowSorter but problem is that modelToView and viewToModel are private arrays and I can't access them so I have problems when rows are filtered...
I am trying to avoid writing my own RowSorter implementation because it will take time to do it right...
UPDATE 2013-10-07:
I have accomplished this by implementing my own RowSorter. After experimenting with both approaches I realised the second one gives me more flexibility.
The "right" way to get a JTable that uses a RowSorter to sort correctly is to override AbstractTableModel.getColumnClass(int). Then you can call DefaultRowSorter.setSortKeys() to specify the sort order.
These methods will allow you to add/delete rows to/from your model in any order and be satisfied they will be displayed in the "correct" sorted order.
If you need special sorting, i.e. sorting based on something other than the objects' compareTo() methods, you can call DefaultRowSorter.setComparator() to provide your own comparison function to work on that column.
I'm not sure what you mean by "cursor", but if you mean that any newly-inserted row should be selected, that would be accomplished by calling JTable.setRowSelectionInterval(), which takes view indices, so they need to be converted (e.g. convert your model index, possibly the last row in the model that you just inserted, into a view index).
Not sure if I should have just edited my original answer but I think it's completely wrong now.
I think your first solution is the best. I wouldn't be worried about inserting data in the table model being costly until I saw it first-hand (or if you knew you had hundreds of thousands of rows or something).
So - something like this is what I'd go for:
int modelIndex = table.convertRowIndexToModel(table.getSelectedRow());
tableModel.insertRow(modelIndex, myRowVector);
Glancing at the DefaultTableModel code, it looks like this might be a fairly fast method, though there's probably no way around the Vector.insertElementAt() overhead... I would be willing to bet it's a lower overhead altogether than using a RowSorter at all.
I am using a JTable in java for listing the values from database.
I need something like, I need to list few set of values in the JTable. And when we scroll down or scroll up using scroll pane of JTable, next set of values must be loaded from database. so that instead of loading all values, i can list few values and scrolling action will retrieve next range of values.
How can I do this?
Can any one suggest me an idea for this?
You need to implement a TableModel, like subclassing DefaultTableModel. Need to track which rows you already retrieved from database and when table request the table model more rows, grab from database. This will occurs as user scrolls the table. Be careful if the total number of rows is huge, as you should discard rows retrieved and not shown currently in the table to save memory.
This question is not easy and implementations could vary. You can maintain an open connection and read from the ResultSet, you can open a new connection each time... If retrieving the rows takes too much time, the user will experience the scroll "freezes" while retrieving the new rows. Another problem is calculating the total number of rows, as table needs it for several operations (calculate the scrolling available...), implementing getRowCount() could be problematic with very large queries. You should perform a SELECT COUNT before to get the total number of rows and then perform the SELECT to start grabbing they.
Mi advice is to retrieve all rows from database and pass to the table. If the total number of rows is not too much large (<10000), JTable handles without problem such number of rows (some memory is required!) and the scroll will be perfectly smooth and only one database access is performed. If you know user already will not scroll the entire set of rows, try to adopt another technique, like limiting the total number of rows returned from database, setting a filter and getting all the rows (limited for example to 100) and show all in the table in order to avoid memory usage and database access.
In the days of Java 1.4.1 and Pentium IV (2004), we use to populate the JTable with all rows returned from database, limiting they to 10000 (WHERE ROWNUM < 10000 in Oracle) and the application needs more memory but works fine and smooth. The time required to retrieve all the rows was larger, but use a waiting dialog and let the user wait for the data (or he/she can filter the data to retrieve fewer rows).
I am not sure how to do it with the scroll action. You could have next and back buttons instead. You could initially show the first set of values. Clicking on next or back will cause the next set to be retrieved. You might need some reference to the first and last value in the displayed list.