What happens upon executing statement? does it retrieves all the rows into memory?Where the rows are stored in the Resultset, and how they are fetched into java program?
Update
On calling resultSet.next(); Does it goes to database fetch a singleRow come back at java side and display it? and ResultSet has cursor is it similar to database cursor?
A common misunderstanding is that the ResultSet must be some kind of container that holds all the rows from the query, so that people try to pass it around the application and are surprised that it becomes invalid when the database connection used to create it closes. The ResultSet is more like a database cursor, it's something you use to pull rows back from the database. The ResultSet has a fetch size that suggests to the driver how many rows it can get from the server at a time, so that it can retrieve the rows in chunks and buffer them for when they're needed.
It is driver specific. E.g. Postgres JDBC loads all rows by default to memory, Oracle uses cursor on server side and fetches only part of rows.
Related
My problem is related to JDBC queries where the number of records in the Table is huge. The end goal is getting the data from DB in a streamed fashion where by you get the data chunk by chunk.
This is possible by creating Multiple SQL statements by key words such as LIMIT , OFFSET. But in this case there will be multiple DB calls which will cost more time.
Is there a way where by you do not load an entire result Set into memory & can get data in chunks without having additional DB calls?
Thanks
First, if you are getting data in chunks from the database, you will be doing multiple database calls. You won't be executing as many queries.
Second, yes it is possible. There is a DB concept known as a "cursor".
Connection cn = //..
// Very important - JDBC default is to commit after every statement
// which will cause DB to close the cursor
cn.setAutoCommit(false);
Statement st = cn.prepareStatement("SELECT * FROM TBL_FOO");
// Cache 50 rows at the client at a time
st.setFetchSize(50);
ResultSet rs = st.executeQuery();
while (rs.next()) {
// Move the cursor position forward - moving past cached rows triggers another fetch
}
rs.close();
st.close();
Note, the database will have fetched all the rows when it executed the query, and the result set will occupy DB memory until you close the cursor. Remember, the DB is a shared resource.
I have a performance related question. I need to retrieve about 500 rows from the database for the purpose of using the Apache POI to export the results into a Microsoft Excel spreadsheet.
Up until now for all my database queries I have been populating a PL/SQL object in the database layer and then returning that PL/SQL object to the Java and looping through the results.
But now that I need to return such a large result set to the Java from the db layer I've been asked a question about whether or not I think it might be better performance wise to return the 500 rows into the Java via an XML Clob.
This is a bit of an open question but I was hoping to get peoples opinion on this please.
thanks
As per http://docs.oracle.com/cd/E11882_01/java.112/e16548/resltset.htm#JJDBC28621
By default, when Oracle JDBC runs a query, it retrieves a result set
of 10 rows at a time from the database cursor. This is the default
Oracle row fetch size value. You can change the number of rows
retrieved with each trip to the database cursor by changing the row
fetch size value.
The following methods are available in all Statement, PreparedStatement, CallableStatement, and ResultSet objects for setting and getting the fetch size:
void setFetchSize(int rows) throws SQLException
int getFetchSize() throws SQLException
Use a Java ResultSet. This will fetch only some rows at a time, as you need them. Here is an example of how to use it: http://docs.oracle.com/javase/tutorial/jdbc/basics/retrieving.html
Basically, every time you ask for a new row, as in rs.next(), the JDBC system decides if the data is available on the client, or needs to be fetched from the server. This way, you are not fetching all of the data at once.
In my java code, I access an oracle database table with an select statement.
I receive a lot of rows (about 50.000 rows), so the rs.next() needs some time to process all of the rows.
using ResultSet, the processing of all rows (rs.next) takes about 30 secs
My goal is to speed up this process, so I changed the code and now using a CachedRowSet:
using CachedRowSet, the processing of all rows takes about 35 secs
I don't understand why the CachedRowSet is slower than the normal ResultSet, because the CachedRowSet retrieves all data at once, while the ResultSet retrieves the data every time the rs.next is called.
Here is a part of the code:
try {
stmt = masterCon.prepareStatement(sql);
rs = stmt.executeQuery();
CachedRowSet crset = new CachedRowSetImpl();
crset.populate(rs);
while (rs.next()) {
int countStar = iterRs.getInt("COUNT");
...
}
} finally {
//cleanup
}
CachedRowSet caches the results in memory i.e. that you don't need the connection anymore. Therefore it it "slower" in the first place.
A CachedRowSet object is a container for rows of data that caches its
rows in memory, which makes it possible to operate without always
being connected to its data source.
-> http://download.oracle.com/javase/1,5.0/docs/api/javax/sql/rowset/CachedRowSet.html
There is an issue with CachedRowSet coupled together with a postgres jdbc driver.
CachedRowSet needs to know the types of the columns so it knows which java objects to create
(god knows what else it fetches from DB behind the covers!).
It therefor makes more roundtrips to the DB to fetch column metadata.
In very high volumes this becomes a real problem.
If the DB is on a remote server, this is a real problem as well because of network latency.
We've been using CachedRowSet for years and just discovered this. We now implement our own CachedRowSet, as we never used any of it's fancy stuff anyway.
We do getString for all types and convert ourselves as this seems the quickest way.
This clearly wasn't an issue with fetch size as postgres driver fetches everything by default.
What makes you think that ResultSet will retrieve the data each time rs.next() is called? It's up to the implementation exactly how it works - and I wouldn't be surprised if it fetches a chunk at a time; quite possibly a fairly large chunk.
I suspect you're basically seeing the time it takes to copy all the data into the CachedRowSet and then access it all - basically you've got an extra copying operation for no purpose.
Using normal ResultSet you can get more optimization options with RowPrefetch and FetchSize.
Those optimizes the network transport chunks and processing in the while loop, so the rs.next() has always a data to work with.
FetchSize has a default set to 10(Oracle latest versions), but as I know RowPrefetch is not set. Thus means network transport is not optimized at all.
I have a .jsp page where I have a GUI table that displays records from an Oracle database. This table allows typical pagination behaviour, such as "FIRST", "NEXT", "PREVIOUS" and "LAST". The records are obtained from a Java ResultSet object that is returned from executing a SQL statement.
This ResultSet might be very big, so my question is:
If I have a ResultSet containing one million records but my table only displays the data from the first ten records in the ResultSet, is the data only fetched when I start requesting record data or does all of the data get loaded into memory entirely once the ResultSet is returned from executing a SQL statement?
The Java ResultSet is a pointer (or cursor) to the results in the database. The ResultSet loads records in blocks from the database. So to answer your question, the data is only fetched when you request it but in blocks.
If you need to control how many rows are fetched at once by the driver, you can use the setFetchSize(int rows) method on the ResultSet. This will allow you to control how big the blocks it retrieves at once.
The JDBC spec does not specify whether the data is streamed or if it is loaded into memory. Oracle streams by default. MySQL does not. To get MySQL to stream the resultset, you need to set the following on the Statement:
pstmt = conn.prepareStatement(
sql,
ResultSet.TYPE_FORWARD_ONLY,
ResultSet.CONCUR_READ_ONLY);
pstmt.setFetchSize(Integer.MIN_VALUE);
The best idea is make a sub query and display 100 or 1000 rows at a time/in single page. And managing the connection by connection pooling.
To make a sub query you can use Row count in oracle and Limit in MY SQL.
While the JDBC spec does not specify whether or not the all data in the result set would get fetched, any well-written driver won't do that.
That said, a scrollable result set might be more what you have in mind:
(link redacted, it pointed to a spyware page)
You may also consider a disconnected row set, that's stored in the session (depending on how scalable your site needs to be):
http://java.sun.com/j2se/1.4.2/docs/api/javax/sql/RowSet.html
lets say we have a table that contains 500 records in it
PreparedStatement stm=con.prepareStatement("select * from table");
stm.setFetchSize(100);// now each 100 records are loaded together from the database into the memory,
// and since we have 500 5 server round trips will occur.
ResultSet rs = stm.executeQuery();
rs.setFetchSize (50);//overrides the fetch size provided in the statements,
//and the next trip to the database will fetch the records based on the new fetch size
I'm running tomcat and have some jsp pages that display a subset of a table. I show 20 rows at a time on a single page. When the table has large amounts of data, the jsp page doesn't render. I'm guessing that the ResultSet is using a client side cursor. I've worked with ASP in the past, and we always used server side forward only cursors, and never had any problems with large amounts of data. Our database is oracle 10g.
How can I specify a server-side forward-only cursor in JDBC?
The oracle driver implements server-side cursors via the FetchSize property.
Unfortunately, JDBC doesn't explicitly allow for setting client vs server-side cursors, so different drivers implement it in different ways. Here are the other links that helped:
Fetch Size
Cursors
Oracle Driver
Statement stmt = con.createStatement(ResultSet.TYPE_FORWARD_ONLY);
ResultSet rs = stmt.executeQuery(sql);
This should set it but apparently some drivers ignore it.
You could always try and set it again at ResultSet level.
rs.setFetchDirection(ResultSet.TYPE_FORWARD_ONLY);
Hope that helps.
Not quite answering the question, but have you considered explicitly adding paging to your SELECT query using ROWNUM or ROWNUMBER in your WHERE clause?
eg: for the second page of data, 20 element page size:
SELECT *
FROM MyDataObjects
WHERE rownum > 20 AND rownum < 41
This would ensure that at most one page of records are returned, removing the large cursor issue.