I have some trivial table in database (let say Oracle10g) and I need to implement at DAO ability to delete multiple records. The method remove() receives as a parameter an array of ids (integers).
For now I have a query string "DELETE FROM news WHERE id = ?" which I use at PreparedStatement. I simply add batch for every id from array and then perform execute on PreparedStatement.
I wonder if there any ability to perform it through one query statement, something like "DELETE FROM news WHERE id IN ?". But I cannot find how to properly set an array of integers instead of '?'.
The same question applies to Hibernate and JPA. If there any constructions to solve this ? Because now I use batch-like-way: add Query to Session on every id from array and commit transaction.
The best I've seen done is to dynamically build the String used by the PreparedStatement, inserting the proper # of ?, sequences, then use a for loop to call setInt as appropriate for each row - each row to be deleted in your case.
JPA provides a special syntax for this (can accept a Collection to populate the list of arguments), since it has to create the SQL anyway - and likely does so very similar to how I just described. Specifics as to the API calls (for both JPA and HQL) are available at Hibernate HQL Query : How to set a Collection as a named parameter of a Query? .
Related
I have a JPA method in my repository trying to find entities with a where clause. The problem is that i have huge data set, and when i try to send more than 32k elements in the list clause, i received an error. I found that is a PostgreSQL driver limitation, but i cant find a workaround.
I tried Pageable request but is hard to send only 30k for 8 millions record. Is there any possibility to send more than 30k objects in my in list where clause?
List<Object> findAllByIdIn(List<Long> ids)
No, you don't want to do it especially if you plan to send 8 million identifiers. Working around the IN statement or bind parameter limit is inefficient. Consider the following:
Thousands of bind parameters will result in megabytes of SQL. It will take considerable time to send the SQL text to the database. In fact the database might take longer to read the SQL text than execute the query as per Tom's answer to "Limit and conversion very long IN list: WHERE x IN ( ,,, ...)" question.
SQL parsing will be inefficient. Not only the megabytes of SQL text take time to read but with increased bind parameter count each query will usually have a distinct number of bound parameters used. This distinct bound parameter count is going to result in each query being parsed and planned separately (see this article which explains it).
There is a hard limit of bound parameters in a SQL statement. You just discovered it, 32760.
For those types of queries it's usually better to create temporary tables. Create a new temporary table before your query, insert all the identifiers into it and join it with the entity table. This join will be equivalent to IN condition except SQL text will be short.
It's important to understand from where are these 8 million identifiers loaded. If you are pulling these from the database in the previous query just to pass them back to the the next query you most likely want to write a stored procedure. There is possibly a flaw in your current approach, JPA is not always the right tool for the job.
Is it possible to nest an Update query within a Select query?
Or is there another method of updating rows given that the first word in the query must be 'Select' and ensuring that it is one query (ie. there is no semicolon between queries).
Furthermore, it must be accepted by JdbcTemplate query (which is why there must be no semicolons).
Thanks for the help.
=====
Update:
I'm trying to inject SQL to update table rows. The table structure doesn't matter but the table must already exist and the rows that it is trying to update must already exist as well.
Example query:
UPDATE users
SET username = 'Mike';
The only requirement is that the first word in the SQL query must be SELECT because that is the only basic sanity check that the server does. Is it possible to do something like this?
=====
Update:
One solution mentioned was to use PL/SQL to call an function that would mutate data. I'm currently looking to see if you can create this function and call it in one or more SELECT statement.
I have the following query:
#Select("SELECT* FROM "+MyData.TABLE_NAME+" where data_date = #{refDate}")
public List<MyData> getMyData(#Param("refDate") Date refDate);
This table data is HUGE! Loading so many rows in memory is not the best way!
Is it possible to have this same query return a resultset so that I can just iterate over one item?
edit:
I tried adding:
#ResultType(java.sql.ResultSet.class)
public ResultSet getMyData(#Param("refDate") Date refDate);
but it gives me:
nested exception is org.apache.ibatis.reflection.ReflectionException: Error instantiating interface java.sql.ResultSet with invalid types () or values (). Cause: java.lang.NoSuchMethodException: java.sql.ResultSet.<init>()
I'd suggest you use limit in your query. limit X, Y syntax is good for you. Try it.
If the table is huge, the query will become slower and slower. Then the best way to to iterate will be to filter based on id and use limit.
such as
select * from table where id>0 limit 100 and then
select * from table where id>100 limit 100 etc
There are multiple options you have ...
Use pagination on database side
I will just suppose the database is oracle. However other db vendors would also work. In oracle you have a rownum with which you can limit number of records to return. To return desired number of records you need to prepare a where clause using this rownum. Now, the question is how to supply a dynamic rownum in a query. This is where dynamic sqls of mybatis comes in use. You can pass these rownum values inside a parameter map which there onwards you can use in your query inside a mapper xml using a #{} syntax. With this approach you filter the records on db level itself and only bring or prepare java objects which are needed or in the current page.
Use pagination on mybatis side
Mybatis select method on sqlSession has a Rowbounds attribute. Populate this as per your needs and it will bring you those number of records only. Here, you are limiting number of records on mybatis side whereas in first approach the same was performed on db side which is better performant .
Use a Result handlers
Mybatis will give you control of actual jdbc result set. So, you can do/iterate over the result one by one here itself.
See this blog entry for more details.
Hibernate Criteria support provides a setMaxResults() method to limit the results returned from the db.
I can't find any answer to this in their documentation - how is this implemented? Is it querying for the entire result set and then returning only the request number? Or is it truly limiting the query on the database end (think LIMIT keyword as in mySql).
This is important because if a query could potentially return many many results, I really need to know if the setMaxResults() will still query for all the rows in the database (which would be bad).
Also - if its truly limiting the number of rows on the database end, how is it achieving this cross-db (since I don't think every rdbms supports a LIMIT functionality like mySql does).
Hibernate asks the database to limit the results returned by the query. It does this via the dialect, which uses whatever database-specific mechanism there is to do this (so for SQL Server it will do somthing like "select top n * from table", Oracle will do "select * from table where rownum < n", MySQL will do "select * from table limit n" etc). Then it just returns what the database returns.
The class org.hibernate.dialect.Dialect contains a method called supportsLimit(). If dialect subclasses override this method, they can implement row limit handling in a fashion native to their database flavor. You can see where this code is called from in the class org.hibernate.loader.Loader which has a method titled prepareQueryStatement, just search for the word limit.
However, if the dialect does not support this feature, there is a hard check in place against the ResultSet iterator that ensures Java object (entity) results will stop being constructed when the limit is reached. This code is also located in Loader as well.
I use both Hibernate and Hibernate Search and without looking at the underlying implementation I can tell you that they definitely do not return all results. I have implemented the same query returning all results and then changed it to set the first result and max results (to implement pagination) and the performance gains were massive.
They likely use dialect specific SQL for this, e.g. LIMIT in MySQL, ROWNUM in Oracle. Your entity manager is aware of the dialect that you are using so this is simple.
Lastly if you really want to check what SQL Hibernate is producing for this query, just set the "show_sql" property to true when you create your entity manager / factory and it spits out all the SQL it is running to the console.
HQL does not suppport a limitation inside a query like in SQL, only the setMaxResults() which you also found.
To find out if it transform the setMaxResults() into a LIMIT query, you can turn on your SQL logging.
I know Question is bit old. But yes setMaxResults() is truly limiting the number of rows on the database end.
If you really look into your Hibernate SQL output, you can find the following SQL statement has been appended to your query.
limit ?
I need to get the list of columns in a table using mybatis/ibatis in java 1.5.
That's not a typical requirement (99.99% of applications using iBatis or whatever ORM knows the DB schema). iBatis is a SQL mapper, you must write the SQL query yourself. And there is no standard SQL query (AFAIK) that gives you the number of columns in a table.
I can only suggest two approaches:
Make a SQL query selecting from the catalog tables. That's the normal way of knowing about your DB metadata. But that depends on your particular database engine. And it's not related to iBatis.
QUick and dirty: make an ad-hoc query SELECT * FROM MYTABLE LIMIT 1 (replace LIMIT for your DB analog), map that in iBatis through a HashMap, and in your DAO just count the number of keys.
For Mybatis:You need to use resultType instead of resultmap.
resultType must be of returning collection data type, by knowing the size of collection you can get no. of columns and more over if you are going with HashMap you can get column names too in keys.