What is a good way to extend org.teiid.translator.ws to read a complete set of records by iterating over all pages returned by a paginated webservice?
Since the pagination of results is not part of any REST API standard (unlike OData), you would have to extend the current translator and provide that custom behaviour to scroll through the pagination. Unlike JDBC kind of resultset scrolling, you would need to devise a way to execute the URL with your offsets each time the Teiid engine asks for next batch of results. If you want an example take look at OData translator for similar flow.
Related
I have an Saleforce app that allows me to execute REST API calls, and I need to retrieve orders (/services/data/v47.0/sobjects/Order) by status.
I've found some manual that describes similar filtering on another entitiy (https://developer.salesforce.com/docs/atlas.en-us.api_placeorder.meta/api_placeorder/sforce_placeorder_rest_api_standalone.htm).
However when trying to execute followin request, it seems that all statuses returned:
GET /services/data/v47.0/sobjects/Order?order.status='ddd'
I also tried some variations of query params. Is this functionality supported?
/sobjects service will let you learn dynamically what fields (standard and custom) exist in Order table (or any other really), what types they are, picklist values...
To retrieve actual data you can use query resource. (Salesforce uses a dialect of SQL, called SOQL. If you've never used it before it'll look bit weird the moment you want to do any JOINs, would be nice if a SF developer would fill you in)
This might be a good start
/services/data/v47.0/query/?q=SELECT Id, Name, OrderNumber FROM Order WHERE Status = 'Draft' LIMIT 10
Never seen the API you've linked to, interesting stuff. But I don't see anything obvious that would let you filter by status there so the more generic "query anything you wish" might work better for you. Play a bit and perhaps https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/dome_query.htm will suit your needs more?
I am supposed to join some huge SQL tables with the json of some REST services by some common key ( we are talking about multiple sql tables with a few REST services calls ). The thing is this data is not real time/ infinite stream and also don’t think I could order the output of the REST services by the join columns. Now the silly way would be to bring all data and then match the rows, but that would imply to store everything in memory/ some storage like Cassandra or Redis.
But, I was wondering if flink could use some king of stream window to join say X elements ( so really just store in RAM just those elements at a point ) but also storing the nonmatched element for later match in maybe some kind of hash map. This is what I mean by smart join.
The devil is in the details, but yes, in principle this kind of data enrichment is quite doable with Flink. Your requirements aren't entirely clear, but I can provide some pointers.
For starters you will want to acquaint youself with Flink's managed state interfaces. Using these interfaces will ensure your application is fault tolerant, upgradeable, rescalable, etc.
If you wanted to simply preload some data, then you might use a RichFlatmap and load the data in the open() method. In your case a CoProcessFunction might be more appropriate. This is a streaming operator with two inputs that can hold state and also has access to timers (which can be used to expire state that is no longer needed, and to emit results after waiting for out-of-order data to arrive).
Flink also has support for asynchronous i/o, which can make working with external services more efficient.
One could also consider approaching this with Flink's higher level SQL and Table APIs, by wrapping the REST service calls as user-defined functions.
I have found the Jquery datatables plug in extremely useful for simple, read only applications where I'd like to give the user pagination, sorting and searching of very large sets of data (millions of rows using server side processing).
I have a system for reusing this code but I end up doing the same thing over and over alot. I'd like to write a very generalized api that I essentially just need to configure the sql needed to retrieve the data used in the table. I am looking for a good design pattern/approach to do this. I've seen articles like this http://www.codeproject.com/Articles/359750/jQuery-DataTables-in-Java-Web-Applications and have a complete understanding of how server side processing works (have done it in java and asp.net many times). For someone to answer you will probably need to have a deep understanding of how server side processing works in java but here are some issues that come up with attempting to do this:
I generally run three separate queries. A count without the search clause, a count with the clause included, the query for the actual data. I haven't found an efficient way to do all 3 at once and doing so requires a lot of extra data to come back from db (ie counts over and over). The api needs to support behavior based on these three different queries and complex queries at that. I generally row number () over an index for the pagination to be relatively speedy with large data.
*where clause changes dynamically (user can search over a variable number of rows).
*order by clause changes for the same reason.
overall, each case is often pretty specific to the data we need. Is there a good way to abstract this so that I can do minimal work when I want to use the plug in server side.
So, the steps are as follows in most projects:
*extract the params the plug on sends to the server (alot of times my own are added, mostly date ranges)
*build the unfiltered count query (this is rarely dynamic).
*build the filtered count query (is dynamic)
*build the data query
*construct a model object of the table and return it as json.
A lot of the issues occur setting the prepared statements with a variable number of parameters. Dynamically generating the sql in a general way (say based on just column names) seems unlikely. I am wondering if someone else has created something they are using for this or if it sounds like a specific pattern is applicable. It has just occurred to me that creating a reusable filter may be helpful in java. Any advice would be greatly appreciated. Feel free to be language agnostic as the architecture is what I'm trying to figure out.
We have base search criteria where all request parameters relevant to DataTables are mapped onto class properties (fields) and custom search criteria class that extends base and contains specific to business logic fields for sutom search. Also on server side we have repository class that takes custom search criteria as an argument and makes queries to database.
If you are familiar with C#, you could check out custom binding code and example of usage.
You could do such custom binding in your Java code as well.
There have been several Play Framework pagination questions but all have been using JPA or ebean. I need to paginate data I am getting returned from a web service. Is there a way to do this with the Play Pagination Module or am I stuck with jQuery? I am also new to play and Java coming from asp.net MVC. The web service is returning a List of whatever model I am querying.
You should not paginate results from web service in the Play's controller as it would be not optimal, consider 3 scenarios (in that order)
Let's say that you want to display 10 items at once, but your generator returns 100.000 for sample query (which means 10.000 of pages)
Pagination should be done by data generator (web service in this case), so you should sent a query containg data: what are you looking for, how big page you want to get and which page you need ie: ?q=pagination&size=10&page=123. This will respond with List of 10 items on page 123. If you have possibility to change the web service to add pagination it's best choice.
If web service doesn't offer pagination get all results at once and use jQuery to paginate it. So if you'll get set of 100.000 items put all of them to the client and make sure that every page change won't cause querying the web service. Poor option, but still better than next one.
You can iterate returned results to split it easily to the pages, count total amount, etc, etc. If you'll use controller for paginating it, at every page change it will be fetching 100.000 items and then splitting to display just 10 of them. Drama :)
So if you can't use first option and don't want to use jQuery make some caching of the results on your server (ie, by storing results in the database as a separate rows) - in such case you will be able to use Ebean for local searching and paging.
If have a large amount tabular data that I'm trying to display in a JSF datatable. Are there any any implementations or components out there that can handle database paging and sorting?
I currently have to pull all the rows in the table back and handle paging and sorting client side in JSF. Unfortunately this is not very performant and occasionally causes my app server to run out of memory.
Ideally, this datatable implementation would be able to wrap a JDBC query or a Hibernate query somehow. I'm not stuck on JSF 1.2, I plan on upgrading to 2.0 sometime soon if this matters.
Check out richfaces datatable datatable with table filtering and table sorting
Or the extended datatable
But, yes I forgot one small note :), If you need to go to the database and handle your filtering/sorting on db level you will need to provide your own implementation of DataModel by extending the org.ajax4jsf.model.SerializableDataModel.
There are some blogs going that way, see this one
You will always have memory problems when the Java code hauls/copies the entire dataset of a datastore (e.g. a RDBMS) in Java's memory and then do the sorting and filtering right in Java's memory using Java code. It will get worse when you even store it in the session scope of a webapplication.
The most memory efficient approach is to let the DB do the task where it is invented for. The SQL language offers you under each the ORDER BY clause to do the sorting, the WHERE clause to do the filtering and the (DB vendor specific) LIMIT/OFFSET clauses/subselects/functions to return only a subset of records based on firstrow and rowcount or lastrow. This way you end up with only the dataset in Java's memory which is actually to be displayed.
There is no standard JSF component which does exactly that. It will always require the entire dataset being available in the Java memory, because the filtering and sorting needs to happen with pure Java code or Javascripts. JSF knows nothing about SQL, you'll need to provide a custom implementation of javax.faces.model.DataModel to the datatable and/or control/manage that in the data access layer yourself. You can get a lot of new insights/ideas and find a kickoff example in this article. You can also find examples of the needed SQL queries in this JSP-targeted answer I posted before here.
Good luck.
Try this example:
http://weblogs.java.net/blog/caroljmcdonald/archive/2007/05/pagination_of_d.html