I am using JPA(Eclipselink) to execute SQL Server Stored Procedure which returns multiple Resultsets.
As per my knowledge, easiest way to call a SP is:
entityManager.createNativeQuery("exec sp_name").getResultList();
After executing the SP I can only read the single (or very first) ResultSet.
Can some one please suggest how do I retrieve the next ResultSets (or ResultLists())?
I can't answer for EclipseLink specifically, and I'm not sure what the JPA spec says, but most features of JPA took their cue from Hibernate, and Hibernate's limitations on stored procedures are:
The procedure must return a result set. Note that since these servers can return multiple result sets and update counts, Hibernate will iterate the results and take the first result that is a result set as its return value. Everything else will be discarded.
My guess is that JPA defines the same limitation.
EclipseLink has extended support for stored procedures through its StoreProcedureCall class and NamedStoredProcedureCallQuery annotation. You can create a JPA Query using a StoredProcedureCall using the JpaEntityManager interface createQuery(Call) API.
StoreProcedureCall provides additional support over JPA native SQL queries including support for in, out and intout parameters and cursored output parameters and typing. StoreProcedureCall supports calls with both a result set and output parameters, but does not currently support multiple result sets.
What is being returned in your second result set, and how do you want the result returned? You could subclass and customize your SQLServerPlatform in EclipseLink and overwrite the executeStoredProcedure() method to process multiple results sets. It should not be to hard, and you could contribute the code back to EclipseLink if successful. Or you could log and enhancement request for this feature. Looking at the code it should be simple enough to implement, the bigger issue is how to return the multiple result sets.
Related
Looking at the following code:
mongoOps.getCollection("FooBar")
.distinct("_id", query(where("foo").is("bar")).limit(10).getQueryObject());
I would expect this to return only the first 10 distinct _ids of collection FooBar.
But unfortunately, running this against a Collection having more than 10 documents matching the criteria, it returns all of them ignoring the limit(10) specified here.
_id is an ObjectId.
How can I achieve this?
Is it a bug in Spring Data?
I'm already aware how I can achieve this using an aggregate but I'm trying to simplify the code if possible since using an aggregate takes many more lines of code.
FYI: I'm using Spring Data Mongodb 1.10.10 and unfortunately, updating is currently not an option.
I find the existing Spark implementations for accessing a traditional Database very restricting and limited. Particularly:
Use of Bind Variables is not possible.
Passing the partitioning parameters to your generated SQL is very restricted.
Most bothersome is that I am not able to customize my query in how partitioning takes place, all it allows is to identify a partitioning column, and upper / lower boundaries, but only allowed is a numeric column and values.
I understand I can provide the query to my database like you do a subquery, and map my partitioning column to a numeric value, but that will cause very inefficient execution plans on my database, where partition pruning (true Oracle Table Partitions), and or use of indexes is not efficient.
Is there any way for me to get around those restriction ... can I customize my query better ... build my own partition logic. Ideally I want to wrap my own custom jdbc code in an Iterator that I can be executed lazily, and does not cause the entire resultset to be loaded in memory (like the JdbcRDD works).
Oh - I prefer to do all this using Java, not Scala.
Take a look at the JdbcRDD source code. There's not much to it.
You can get the flexibility you're looking for by writing a custom RDD type based on this code, or even by subclassing it and overriding getPartitions() and compute().
I studied both JdbcRDD and new Spark SQL Data source API. None of them support your requirements.
Most likely this will be your own implementation. I recommend writing new Data sources API instead of sub-classing JdbcRDD which became obsolete in Spark 1.3.
I am using Hiberante to connect to postgres database. I am trying to insert a record into the database. I have the values for the record in a string array which I got from a csv file. This is my dao code
StringBuffer query=new StringBuffer("insert into t_wonlist values(");
for(int i=0;i<67;i++){
query.append(values[i]+",");
}
query.deleteCharAt(query.lastIndexOf(","));
query.append(");");
sessionfactory.getCurrentSession().createSQLQuery(query.toString()).executeUpdate();
System.out.println("Query executed");
sessionfactory.getCurrentSession().flush();
I am using StringBuffer, so that I can append the values into the query using a for loop.
but when I execute the query I am getting the following exception
org.postgresql.util.PSQLException: No value specified for parameter 1.
I am sure that the number of parameters is correct. Can someone help me. Thanks
You're approaching this in a bizarre and backwards manner.
The immediate problem is probably failure to escape/quote a ? in one of the input strings, so PgJDBC thinks it's a query parameter. That doesn't mean you should fix it by escaping/quoting question marks, it's a sign you're taking entirely the wrong approach.
Please read this page on SQL injection and this site.
You're using the Hibernate ORM, so you'd usually be using the JPA interface or the direct Hibernate interface to create new domain objects and persisting them. The typical approach is to new an object, then use the EntityManager.persist method (if using JPA) or the Session.save method (if using Hibernate directly) to persist the entities.
If you want to use direct JDBC instead you should be creating a JDBC PreparedStatement, setting its parameters, and then applying it. See this tutorial. Since you're loading CSV you'd usually do this in a JDBC batch, though this doesn't actually gain you much in PostgreSQL.
Better yet, since you're importing CSV you can probably just use PostgreSQL's built-in COPY command via PgJDBC's CopyManager to stream the changes efficiently into the target table.
Hibernate Criteria support provides a setMaxResults() method to limit the results returned from the db.
I can't find any answer to this in their documentation - how is this implemented? Is it querying for the entire result set and then returning only the request number? Or is it truly limiting the query on the database end (think LIMIT keyword as in mySql).
This is important because if a query could potentially return many many results, I really need to know if the setMaxResults() will still query for all the rows in the database (which would be bad).
Also - if its truly limiting the number of rows on the database end, how is it achieving this cross-db (since I don't think every rdbms supports a LIMIT functionality like mySql does).
Hibernate asks the database to limit the results returned by the query. It does this via the dialect, which uses whatever database-specific mechanism there is to do this (so for SQL Server it will do somthing like "select top n * from table", Oracle will do "select * from table where rownum < n", MySQL will do "select * from table limit n" etc). Then it just returns what the database returns.
The class org.hibernate.dialect.Dialect contains a method called supportsLimit(). If dialect subclasses override this method, they can implement row limit handling in a fashion native to their database flavor. You can see where this code is called from in the class org.hibernate.loader.Loader which has a method titled prepareQueryStatement, just search for the word limit.
However, if the dialect does not support this feature, there is a hard check in place against the ResultSet iterator that ensures Java object (entity) results will stop being constructed when the limit is reached. This code is also located in Loader as well.
I use both Hibernate and Hibernate Search and without looking at the underlying implementation I can tell you that they definitely do not return all results. I have implemented the same query returning all results and then changed it to set the first result and max results (to implement pagination) and the performance gains were massive.
They likely use dialect specific SQL for this, e.g. LIMIT in MySQL, ROWNUM in Oracle. Your entity manager is aware of the dialect that you are using so this is simple.
Lastly if you really want to check what SQL Hibernate is producing for this query, just set the "show_sql" property to true when you create your entity manager / factory and it spits out all the SQL it is running to the console.
HQL does not suppport a limitation inside a query like in SQL, only the setMaxResults() which you also found.
To find out if it transform the setMaxResults() into a LIMIT query, you can turn on your SQL logging.
I know Question is bit old. But yes setMaxResults() is truly limiting the number of rows on the database end.
If you really look into your Hibernate SQL output, you can find the following SQL statement has been appended to your query.
limit ?
My goal is actually to dump all the data of a database to an XML file. The database is not terribly big, it's about 300MB. The problem is that I have a memory limitation of 256MB (in JVM) only. So obviously I cannot just read everything into memory.
I managed to solve this problem using iBatis (yes I mean iBatis, not myBatis) by calling it's getList(... int skip, int max) multiple times, with incremented skip. That does solve my memory problem, but I'm not impressed with the speed. The variable names suggests that what the method does under the hood is to read the entire result-set skip then specified record. This sounds quite redundant to me (I'm not saying that's what the method is doing, I'm just guessing base on the variable name).
Now, I switched to myBatis 3 for the next version of my application. My question is: is there any better way to handle large amount of data chunk by chunk in myBatis? Is there anyway to make myBatis process first N records, return them to the caller while keeping the result set connection open so the next time the user calls the getList(...) it will start reading from the N+1 record without doing any "skipping"?
myBatis CAN stream results. What you need is a custom result handler. With this you can take each row separately and write it to your XML file. The overall scheme looks like this:
session.select(
"mappedStatementThatFindsYourObjects",
parametersForStatement,
resultHandler);
Where resultHandler is an instance of a class implementing the ResultHandler interface. This interface has just one method handleResult. This method provides you with a ResultContext object. From this context you can retrieve the row currently being read and do something with it.
handleResult(ResultContext context) {
Object result = context.getResultObject();
doSomething(result);
}
No, mybatis does not have full capability to stream results yet.
EDIT 1:
If you don't need nested result mappings then you could implement a custom result handler to stream results. on current released versions of MyBatis. (3.1.1) The current limitation is when you need to do complex result mapping. The NestedResultSetHandler does not allow custom result handlers. A fix is available, and it looks like is currently targeted for 3.2. See Issue 577.
In summary, to stream large result sets using MyBatis you'll need.
Implement your own ResultSetHandler.
Increase fetch size. (as noted below by Guillaume Perrot)
For Nested result maps, use the fix discussed on Issue 577. This fix also resolves some memory issues with large result sets.
I have successfully used MyBatis streaming with the Cursor. The Cursor has been implemented on MyBatis at this PR.
From the documentation it is described as
A Cursor offers the same results as a List, except it fetches data
lazily using an Iterator.
Besides, the code documentation says
Cursors are a perfect fit to handle millions of items queries that
would not normally fits in memory.
Here is an example of implementation I have done and which I was able to successfully use it:
import org.mybatis.spring.SqlSessionFactoryBean;
// You have your SqlSessionFactory somehow, if using Spring you can use
SqlSessionFactoryBean sqlSessionFactory = new SqlSessionFactoryBean();
Then you define your mapper, e.g., UserMapper with the SQL query that returns a Cursor of your target object, not a List. The whole idea is to not store all the elements in memory:
import org.apache.ibatis.annotations.Select;
import org.apache.ibatis.cursor.Cursor;
public interface UserMapper {
#Select(
"SELECT * FROM users"
)
Cursor<User> getAll();
}
Then you write the that code that will use an open SQL session from the factory and query using your mapper:
try(SqlSession sqlSession = sqlSessionFactory.openSession()) {
Iterator<User> iterator = sqlSession.getMapper(UserMapper.class)
.getAll()
.iterator();
while (iterator.hasNext()) {
doSomethingWithUser(iterator.next());
}
}
handleResult receives as many records as the query gets, no pause.
When there are too many records to process I used sqlSessionFactory.getSession().getConnection().
Then as, normal JDBC, get a Statement, get the Resultset, and process one by one the records. Don't forget to close the session.
If just dumping all the data without any ordering requirement from tables, why not directly do the pagination in SQL? Set a limit to the query statement, where specifying different record id as the offset, to separate the whole table into chunks, each of which could directly be read into memory if the rows limit is a reasonable number.
The sql could be something like:
SELECT * FROM resource
WHERE "ID" >= continuation_id LIMIT 300;
I think this could be viewed as an alternative solution for you to dump all the data by chunks, getting rid of the different feature problems in mybatis, or any Persistence layer, support.