I was trying to understand how PagingState works with Statement in Cassandra. I tried with a sample that inserts few 1000s of records into database and tried reading the same from DB with fetch size set to 10 and using paging state. This is working perfectly fine. Here is my sample junit code:
#Before
public void setup() {
cassandraTemplate.executeQuery("create table if not exists pagesample(a int, b int, c int, primary key(a,b))");
String insertQuery = "insert into pagesample(a,b,c) values(?,?,?)";
PreparedStatement insertStmt = cassandraTemplate.getConnection().prepareStatement(insertQuery);
for(int i=0; i < 5; i++){
for(int j=100; j<1000; j++){
cassandraTemplate.executeQuery(insertStmt, new Object[]{i, j, RandomUtils.nextInt()});
}
}
}
#Test
public void testPagination() {
String selectQuery = "select * from pagesample where a=?";
String pagingStateStr = null;
for(int run=0; run<90; run++){
ResultSet resultSet = selectRows(selectQuery, 10, pagingStateStr, 1);
int fetchedCount = resultSet.getAvailableWithoutFetching();
System.out.println(run+". Fetched size: "+fetchedCount);
for(Row row : resultSet){
System.out.print(row.getInt("b")+", ");
if(--fetchedCount == 0){
break;
}
}
System.out.println();
PagingState pagingState = resultSet.getExecutionInfo().getPagingState();
pagingStateStr = pagingState.toString();
}
}
public ResultSet selectRows(String cql, int fetchSize, String pagingState, Object... bindings){
SimpleStatement simpleStatement = new SimpleStatement(cql, bindings);
statement.setFetchSize(fetchSize);
if(StringUtils.isNotEmpty(pagingState)){
statement.setPagingState(PagingState.fromString(pagingState));
}
return getSession().execute(simpleStatement);
}
When I execute this program, I see that every iteration in testPagination is exactly printing 10 records. But here is what the documentation says:
Note that setting a fetch size doesn’t mean that Cassandra will
always return the exact number of rows, it is possible that it
returns slightly more or less results.
I am not really able to understand why Cassandra will return not exactly the same number of rows as specified in fetch size. Is this the case when there is no where clause provided in the query? Will it return exact number of records when a query is constrained on a partition key? Please clarify.
From the CQL protocol specification:
Clients should also not assert that no result will have more than result_page_size results. While the current implementation always respect the exact value of result_page_size, we reserve ourselves the right to return slightly smaller or bigger pages in the future for performance reasons
So it's good practice to always rely on getAvailableWithoutFetching instead of the page size, in case Cassandra changes its implementation in the future.
Related
In a few answers here on SO there is recommended to use ResultSet's function getRow() in order to get the number of entries in a query result. However, following code
private static int getSizeOfResultSet(ResultSet result) throws SQLException{
if(result != null){
int count = 0;
while(result.next()){
count++;
}
System.out.println("Count of resultset is " + result.getRow() + " and count " + count);
return result.getRow();
}else{
System.out.println("Result is null");
return 0;
}
}
is returning me things like
Count of resultset is 0 and count 1
when there is assured that entries are existing (in a JUnit test case). Note that this project has to be in Java6-standard, so maybe this is the issue. Is there anything to consider using the getRow() function?
Additionally, I tried with
result.last();
int c = result.getRow();
result.beforeFirst();
return c;
like mentioned here, but the result is still 0.
EDIT: I should mention that the while(result.next())-loop is for testing purpose. The behaviour without the loop is exactly the same (i.e. 0).
The javadoc states
returns the current row number; 0 if there is no current row
After you've visited all the rows with ResultSet#next() there is no current row.
Use your count value for the number of rows. Or use a SELECT count(*) ... style query.
I'm fairly new to Java and I was wondering how I could get all of the columns of a returned via MySQL query. For instance,
SELECT * FROM `login`
And then I had this code to return the results:
while (rs.next()) {
for (byte i = 1; i < 10; i++) {
result = result+" "+rs.getString(i);
}
}
I want 10 in the for loop to be the maximum number of columns because of dynamic MySQL queries. Is there a simple way to do this? Thanks.
Although, as duffymo suggested, it's better to spell out the columns you want to receive in the query, you can use ResultSet metadata to fetch the number of columns returned:
ResultSetMetaData rsmd = rs.getMetaData();
int numberOfColumns =rsmd.getColumnCount();
while (rs.next()) {
for (byte i = 1; i <= numberOfColumns; i++) {
result = result+" "+rs.getString(i);
}
}
Yes, you'd limit the fields that are returned. That's a SQL issue, not a Java issue. Change your query to something like this:
select this, that, theothercolumn from mytable;
Then you can use this to limit it dynamically:
rs.getMetaData().getColumnCount()
I'd recommend not using SELECT *. If you know the columns you want, spell them out. It'll make your code more robust in the face of column changes and reduce the number of bytes on the wire.
I have a SQL query as shown below.
SELECT O_DEF,O_DATE,O_MOD from OBL_DEFINITVE WHERE OBL_DEFINITVE_ID =?
A collection of Ids is passed to this query and ran as Batch query. This executes for 10000
times for retriveing values from Database.(Some one else mess)
public static Map getOBLDefinitionsAsMap(Collection oblIDs)
throws java.sql.SQLException
{
Map retVal = new HashMap();
if (oblIDs != null && (!oblIDs.isEmpty()))
{
BatchStatementObject stmt = new BatchStatementObject();
stmt.setSql(SELECT O_DEF,O_DATE,O_MOD from OBL_DEFINITVE WHERE OBL_DEFINITVE_ID=?);
stmt.setParameters(
PWMUtils.convertCollectionToSubLists(taskIDs, 1));
stmt.setResultsAsArray(true);
QueryResults rows = stmt.executeBatchSelect();
int rowSize = rows.size();
for (int i = 0; i < rowSize; i++)
{
QueryResults.Row aRow = (QueryResults.Row) rows.getRow(i);
CoblDefinition ctd = new CoblDefinition(aRow);
retVal.put(aRow.getLong(0), ctd);
}
}
return retVal;
Now we had identified that if the query is modified to
add as
SELECT O_DEF,O_DATE,O_MOD from OBL_DEFINITVE WHERE OBL_DEFINITVE_ID in (???)
so that we can reduce it to 1 query.
The problem here is MSSQL server is throwing exception that
Prepared or callable statement has more than 2000 parameter
And were struck here . Can some one provide any better alternative to this
There is a maximum number of allowed parameters, let's call it n. You can do one of the following:
If you have m*n + k parameters, you can create m batches (or m+1 batches, if k is not 0). If you have 10000 parameters and 2000 is the maximum allowed parameters, you will only need 5 batches.
Another solution is to generate the query string in your application and adding your parameters as string. This way you will run your query only once. This is an obvious optimization in speed, but you'll have a query string generated in your application. You would set your where clause like this:
String myWhereClause = "where TaskID = " + taskIDs[0];
for (int i = 1; i < numberOfTaskIDs; i++)
{
myWhereClause += " or TaskID = " + taskIDs[i];
}
It looks like you are using your own wrapper around PreparedStatement and addBatch(). You are clearly reaching a limit of how many statements/parameters can be batched at once. You will need to use executeBatch (eg every 100 or 1000) statements, instead of having it build up until the limit is reached.
Edit: Based on the comment below I reread the problem. The solution: make sure you use less than 2000 parameters when building the query. If necessary, breaking it up in two or more queries as required.
This is the code I am working on:
if(connection.doDatabaseRead(findSQL))
{
ResultSet retRES = connection.getResultSet();
int i = 0;
// did we find anything
while( retRES.next() )
{
//read result from query
suiteNum.add(retRES.getString(i)); // this is the problem
i++;
//let other threads breathe
Thread.yield();
}
}
suiteNum is a string vector
When I try to add the database results to the vector the code crashes with this error.
java.sql.SQLException: Column Index out of range, 0 > 1.
I have the same piece of code working elsewhere in the program but I use real numbers like 0, 1 and 2 instead of i and it works fine.
As I do not know how many results the database request will have I need it to be dynamic but it will only work hard coded.
How can I make it work with i ?
The argument to getString is the column index, not the row index as you seem to think. The function returns the value of the given column in the current row, while next advances the cursor to the next row.
You probably mean:
suiteNum.add(retRES.getString(1));
in which case you can lose i altogether.
Java ResultSet objects are 1-indexed in this regard. The first element is at 1, not 0. See the javadoc.
EDIT: That's true too, but indeed the problem is this appears to be used as a row index! it's certainly the column.
This is your problem:
i = 0;
...
retRES.getString(i);
ResultSet.getString(i) gets a String from column number i
You want something like
while(retRes.next()) {
add(retRes.getString(1);
}
column index starts from 1
As I do not know how many results the database request will have I need it to be dynamic but it will only work hard coded. How can I make it work with i
ResultSetMetaData rsMetaData = rs.getMetaData();
int numberOfColumns = rsMetaData.getColumnCount();
See Also
ResultSetMetaData
Let your i start with 1 as specified in the API docs
if(connection.doDatabaseRead(findSQL))
{
ResultSet retRES = connection.getResultSet();
int i = 1;
// did we find anything
while( retRES.next() )
{
//read result from query
suiteNum.add(retRES.getString(i)); // this is the problem
i++;
//let other threads breathe
Thread.yield();
}
}
I want to get the size of the ResultSet inside the while loop.
Tried the code below, and I got the results that I want. But it seems to be messing up with result.next() and the while loop only loops once if I do this.
What's the proper way of doing this?
result.first();
while (result.next()){
System.out.println(result.getString(2));
System.out.println("A. " + result.getString(5) + "\n" + "B. " + result.getString(6) + "\n" + "C. " + result.getString(7) + "\n" + "D. " + result.getString(8));
System.out.println("Answer: ");
answer = inputquiz.next();
result.last();
if (answer.equals(result.getString(10))) {
score++;
System.out.println(score + "/" + result.getRow());
} else {
System.out.println(score + "/" + result.getRow());
}
}
What's the proper way of doing this?
Map it to a List<Entity>. Since your code is far from self-documenting (you're using indexes instead of column names), I can't give a well suited example. So I'll take a Person as example.
First create a javabean class representing whatever a single row contains.
public class Person {
private Long id;
private String firstName;
private String lastName;
private Date dateOfBirth;
// Add/generate c'tors/getters/setters/equals/hashcode and other boilerplate.
}
(a bit decent IDE like Eclipse can autogenerate them)
Then let JDBC do the following job.
List<Person> persons = new ArrayList<Person>();
while (resultSet.next()) {
Person person = new Person();
person.setId(resultSet.getLong("id"));
person.setFirstName(resultSet.getString("fistName"));
person.setLastName(resultSet.getString("lastName"));
person.setDataOfBirth(resultSet.getDate("dateOfBirth"));
persons.add(person);
}
// Close resultSet/statement/connection in finally block.
return persons;
Then you can just do
int size = persons.size();
And then to substitute your code example
for (int i = 0; i < persons.size(); i++) {
Person person = persons.get(i);
System.out.println(person.getFirstName());
int size = persons.size(); // Do with it whatever you want.
}
See also:
How to check if there is zero-or-one result or one-or-more results and their size
you could do result.last(); and call result.getRow(); (which retrieves the current row number) to get count. but it'll have load the all the rows and if it's a big result set, it might not be very efficient. The best way to go about is to do a SELECT COUNT(*) on you query and get the count like it's demonstrated in this post, beforehand.
This is a tricky question.
Normally, result.last() scrolls to the end of the ResultSet, and you can't go back.
If you created the statement using one of the createStatement or prepareStatement methods with a "resultSetType" parameter, and you've set the parameter to ResultSet.TYPE_SCROLL_INSENSITIVE or ResultSet.TYPE_SCROLL_SENSITIVE, then you can scroll the ResultSet using first() or relative() or some other methods.
However, I'm not sure if all databases / JDBC drivers support scrollable result sets, and there are likely to be performance implications in doing this. (A scrollable result set implies that either the database or the JVM needs to buffer the entire resultset somewhere ... or recalculate it ... and that's expensive for a large resultset.)
The way of getting size of ResultSet, No need of using ArrayList etc
int size =0;
if (rs != null)
{
rs.beforeFirst();
rs.last();
size = rs.getRow();
}
Now You will get size, And if you want print the ResultSet, before printing use following line of code too,
rs.beforeFirst();
There are also another way to get the count from DB.
Note :
This column gets updated when DBA'S do realtime statistics
select num_rows from all_Tables where table_name ='<TABLE_NAME>';