In .NET C#, we used Odata to filter, page, sort the database results from SQL database. Odata in .NET would actually go into the database, and query WHERE, ORDER By Filters to database, instead of extracting all the database results, and applying filtering on the api memory.
I am curious of Java Apache Olingo, queries the database internally or applies filtering on the API memory set.
Resources:
https://www.odata.org/libraries/
https://www.odata.org/documentation/odata-version-2-0/uri-conventions/
Short answer, Olingo 2 JPAProcessor will query the database to only fetch limited data if $filter and $orderby are specified.
TLDR;
There are two things we need to consider here. Apache Olingo is a framework which allows people to implement OData Server and Client in Java, bare bones if will just provide you with the Abstract Classes and Interfaces which you implement OdataProcessors, how ever one wants.
Now coming to the second point there is a JPA Implementation also provided in the Olingo Project, which you can locate here
To answer your question, lets dig into the code of this project to see, how its implemented
We will have to start with JPAProcessorImpl and hop into process(final GetEntitySetUriInfo uriParserResultView), this is where the query and the actuals of how the data is fetched in the database are implemented.
At line 150 the actual query is already built,so if you pass $filter(where clause) and $orderby, we can see the values are actually being passed to the database and are baked in query
Edit: As #Olivier mentions, my original answer referred to OData in .NET and not Olingo.
Unfortunately as you probably have found out yourself, the standard Olingo documentation doesn't answer this particular question directly, though it does mention support for lazy-loading and streaming behaviour (which would not make any sense to filter on the application level).
However, you would be hard-pressed to find an ORM or DB Adapter nowadays that does not support filtering on the database level, as this will almost always be faster than doing so in your application for non-trivial workloads.
I will try to update this answer with a more authorative source.
(original answer)
According to this analysis:
As long as your data provider supports deferred queries and you don't force evaluation by calling something like .ToList(), the query will not be evaluated until the OData filters are applied, and they'll be handled at the database level.
Related
I am working on a project which uses Activiti1 Library and, for these reason, I am using MyBatis API to execute some native queries against Activiti data base.
The problem is: For some builds I am using Oracle and, for other, MySQL database then, as the dialect of this two data bases is different, it would be necessary to use the MyBatis multi vendor support, however, I didn't like it, because I have to deal with specific statments of each type of data base which makes the maintance a hard task, mainly if necessary to add another data base in the future.
So I would like to know if somehow it is possible to use HQL together with MyBatis or if there are another generic SQL engine that can be used in this case.
Or if someone know a free Java API that converts MySQL queries to Oracle queries. I tried to use Hibernate translation, however it didn't work, because Activiti classes is not using JPA, so it is not mapped :(.
Thanks in advance!
Sandro
In Activiti, we do use the multi-vendor support of MyBatis. However, there are only a few cases where we really needed. Do you have that much 'special' queries?
I have couchbase DB deployed in production. I want to write java code to query few details. It doesn't have any views as of today and in order to have views created , I need to go through a lot of process. Is there a way to run queries using code written in couchbase java sdk or is it mandatory get views created to run custom queries.
If you're using Couchbase 4.0 or above, you can use N1QL. Create at least a primary N1QL index once, query anything... You can even create more specific N1QL secondary indexes tailored for queries for which you need better performance.
Views are very specific, they force you to think about exactly how you'll query your data and limit you to that use case. N1QL on the other hand is very general purpose. It's a superset of SQL, with JSON-specific additions.
Of course both work on the assumption that your data is JSON
Without view nor N1QL, you're limited to requests using the keys of documents, which you must know in advance (but that could be a usable alternative nonetheless, eg. if keys are mentioned in another document, or can be reconstructed from the content of another document of which you know the key).
I thought about this solution: get data from web service, insert into table and then join with other table, but it will affect perfomance and, also, after this I must delete all that data.
Are there other ways to do this?
You don't return a record set from a web service. HTTP knows nothing about your database or result sets.
HTTP requests and responses are strings. You'll have to parse out the data, turn it into queries, and manipulate it.
Performance depends a great deal on things like having proper indexes on columns in WHERE clauses, the nature of the queries, and a lot of details that you don't provide here.
This sounds like a classic case of "client versus server". Why don't you write a stored procedure that does all that work on the database server? You are describing a lot of work to bring a chunk of data to the middle tier, manipulate it, put it back, and then delete it? I'd figure out how to have the database do it if I could.
no, you don't need save anything into database, there's a number of ways to convert XML to table without saving it into database
for example in Oracle database you can use XMLTable/XMLType/XQuery/dbms_xml
to convert xml result from webservice into table and then use it in your queries
for example:
if you use Oracle 12c you can use JSON_QUERY: Oracle 12ะก JSON
XMLTable: oracle-xmltable-tutorial
this week discussion about converting xml into table data
It is common to think about applications having a three-tier structure: user interface, "business logic"/middleware, and backend data management. The idea of pulling records from a web service and (temporarily) inserting them into a table in your SQL database has some advantages, as the "join" you wish to perform can be quickly implemented in SQL.
Oracle (as other SQL DBMS) features temporary tables which are optimized for just such tasks.
However this might not be the best approach given your concerns about performance. It's a guess that your "middleware" layer is written in Java, given the tags placed on the Question, and the lack of any explicit description suggests you may be attempting a two-tier design, where user interface programs connect directly with the backend data management resources.
Given your apparent investment in Oracle products, you might find it worthwhile to incorporate Oracle Middleware elements in your design. In particular Oracle Fusion Middleware promises to enable "data integration" between web services and databases.
I work on an application that uses Spring MVC and Hibernate. I am implementing some RESTful web services and am curious how to easily filter collections server side.
As an example, I want to be able to filter a collection of employee entities. I have researched several options, such as RQL, the way Google handles custom searches, Ebay's answer, and even Yahoo's YQL. They all seem to be good answers to the filtering question, but I can not seem to find any libraries that will allow me to easily implement this concept.
I did find here, that:
Apache CXF introduced FIQL support with its JAX-RS implementation since 2.3.0 release
but we are already using Spring MVC.
I'm surprised there is no library for taking the bold query string below, for example, and translating that into SQL or something that Hibernate can use to filter.
/employees?lastname=john OR jon&hiredate lt 20010201
It is entirely possible that I am thinking of this incorrectly, but I wanted to tap into the community's experience and knowledge. What am I missing?
I know this is old, but for completeness, there are at least two libraries that handle parsing RQL:
https://github.com/jazdw/rql-parser
https://github.com/jirutka/rsql-parser (not quite RQL by default, but configurable)
I'm using jazdw/rql-parser myself and started working on an SQL mapper but as Oleksi mentioned there is a lot of custom code required for validating, field mapping, etc. so I don't know how generic I can make it yet.
A library that directly converts a GET like that into SQL could be very insecure. You need to have an intermediate layer to do some validation to make sure that the user isn't messing with the URL to execute a SQL injection.
As far as I know, the best you can do is use your JAX-RS implementation to cleanly read in those query parameters, validate them, and use something like a prepared SQL statement to securely execute them.
I am starting out writing java code and interacting with databases for my "nextbigthing" project. Can someone direct me towards the best way to deal with adding/updating tables/records to databases? Here is my problem. There is too much repitition when it comes to DB code in java. I have to create the tables first (I use mysql). I then create classes in Java for each table. Then I create a AddRow, DeleteRow, UpdateRow and Search* depending on my need. For every table, every need creating this huge ass sql statement and the classes all seems like a huge waste of my time. There has to be a better, easier, more efficient way of doing things. Is there something out there that I do not know that will let me just tell Java what the table is and it automatically generate the queries and execute them for me? Its simple SQL that can be auto generated if it knows the column names and DB table inter dependencies. Seems like a very reasonable thing to have.
Check out Hibernate - a standard Java ORM solution.
User hibernate for mapping your classes to Database.
Set its hbm2ddl.auto to update to avoid writing DDL yourself. But note that this is not the most optimal way to take it to production.
Consider using Hibernate:
https://www.hibernate.org/
It can create java classes with regular CRUD methods from existing database schema.
Of course there is a much better way !
You really want to learn some bits of Java EE, and in particular JPA for database access.
For a complete crash course on Java EE, check out the Sun the Java EE 5 tutorial.
http://java.sun.com/javaee/5/docs/tutorial/doc/
Part 4 - Enterprise Beans
Part 5 - Persistence (JPA)
Then you want to try Hibernate (for instance) which has an implementation of JPA.
This is for Java 5 or later.
If you are still in Java 2, you might want to try Hibernate or iBatis.
You can also try iBatis, if you want control over SQL. Else JPA is good.
You can also try using Seam Framework. It has good reverse-engineering tools.
There is also torque (http://db.apache.org/torque/) which I personally prefer because it's simpler, and does exactly what I need.
With torque I can define a database with mysql(Well I use Postgresql, but Mysql is supported too) and Torque can then query the database and then generate java classes for each table in the database. With Torque you can then query the database and get back Java objects of the correct type.
It supports where clauses (Either with a Criteria object or you can write the sql yourself) and joins.
It also support foreign keys, so if you got a User table and a House table, where a user can own 0 or more houses, there will be a getHouses() method on the user object which will give you the list of House objects the user own.
To get a first look at the kind of code you can write, take a look at
http://db.apache.org/torque/releases/torque-3.3/tutorial/step5.html which contains examples which show how to load/save/query data with torque. (All the classes used in this example are auto-generated based on the database definition).
Or, if Hibernate is too much, try Spring JDBC. It eliminates a lot of boilerplate code for you.
iBatis is another good choice, intermediate between Spring JDBC and Hibernate.
It's just a matter of using the right tools. Use an IDE with tools to autogenerate the one and other.
If you're using Eclipse for Java EE and decide to head to JPA, then I can recommend to take benefit of the builtin Dali plugin. There's a nice PDF tutorial out at Eclipse.org.
If you're using Eclipse for Java EE and decide to head to "good ol" Hibernate, then I can recommend to take benefit of the Hibernatetools plugin. There's good reference guide out at Hibernate.org.
Both tools are capable of reverse-engineering from a SQL table to fullworthy Javabeans/entities and/or mapping files. It really takes most of boilerplate pains away. The DAO pattern is slightly superflous when grabbing JPA. In case of Hibernate you can consider to use a Generic DAO.