I have an web application with Javascript based UI and Jersey based REST service. The UI has certain features requiring complex search criteria to retrieve the data. The UI defines the search criteria with various parameters but the most significant to this discussion is this. A user can create a filter like this-
1. ( Receiver_Email_ID CONTAINS xyz#abc.com AND
2. Sender_Email_ID CONTAINS blocked#abc.com ) OR
3. ( Origin_Domain IS BADDOMAIN.com AND
4. Email_has_attachment == true )
The numbers 1 to 4 just indicate that these are different rows in the UI and that the sequence matters here. The UI represents it in JSON format which on the server could be received as a POJO.
I need the ability to store this criteria such that the UI can be repopulated with the filter and it could be executed. Along with several other tables, I plan to use the following database table to store this-
rule_id
rule_seq
op_paren
left_opr
operator
right_opr
cl_paren
and_or
Does it look right or I can improve it? The other option I have is to store the entire JSON in an embedded document DB such as OrientDB.
From formal point of view each search criteria is Abstract syntax tree It's don't fit you structure. You idea about saving search in OrientDB is very fit to task from this point of view but realization not simple.
Related
I am faily new to MarkLogic (and noSQL) and currently trying to learn the Java API client. My question on searching, which returns back search result snippets / matches, is it possible for the search result to include specific fields in the document?
For example, given this document:
{"id":"1", "type":"classified", "description": "This is a classified type."}
And I search using this:
QueryManager queryMgr = client.newQueryManager();
StringQueryDefinition query = queryMgr.newStringDefinition();
query.setCriteria("classified");
queryMgr.search(query, resultsHandle);
How can I get the JSON document's 3 defined fields (id, type, description) as part of the search result - so I can display them in my UI table?
Do I need to hit the DB again by loading the document via URI (thus if I have 1000 records, that means hitting the DB again 1000 times)?
You have several options to retrieve specific fields with your search results. You could use the Pojo Data Binding Interface. You could read multiple documents matching a query which brings back the entirety of each document which you can then get as a pojo or String or any other handle. Or you can use the same API you're using above but add search options to allow you to extract a portion of a matching document.
If you're bring back thousands of matches, you're probably not showing all those snippets to end users, so you should probably disable snippets using something like
<transform-results apply="empty-snippet" />
in your options.
I have a database with 20,000 records. Each record has a name. When a user wants to view a record, he can visit a webapp and type the name of the record in an inputfield. While typing, results from the database would be shown/filtered matchin what the user typed. I would like to know the basic architecture/concepts on how to program this
I'm using the following language stack:
frontend: html5/javascript (+ajax to make instant calls while user is typing)
backend: java + jdbc to connect to simple sql database
My initial idea is:
A user types text
Whenever a character is entered or removed in the inputfield, make an ajax request to the backend
The backend does a LIKE %input% query on the name field in the database
All data found by the query is send as a json string to the frontend
The frontend processes the json string and displays whatever results it finds
My two concerns are: the high amount of ajax requests to process, in conjunction with the possibly very heavy LIKE queries. What are ways to optimize this? Only search for every two characters they type/remove? Only query for the first ten results?
Do you know of websites that utilise these optimizations?
NOTE: assume the records are persons and names are like real people names, so some names are more common than others.
You can choose SPA approach - load all 20 000 names/ids to client side and then filter it in memory - it's supposed to be the fastest way with minimal load to the database and back-end
Here are possible solutions:
Restirct search to prefix search - LIKE 'prefix%' can be executed efficiently using BTREE-type index.
Measure performance of naive LIKE '%str%' solution - it you are working on B2B application, database will likely load that table in memory and do queries fast enough.
Look at documentation for your database - there could be special features for that like inverted index
as #Stepan Novikov suggested, load your data in memory and search manually
Use specialized search indexers like SOLR or ElasticSearch (likely overkill for only 20k records)
If you are feeling ninja, implement your own N-gram index.
Lets say I wanted a web page that would represent a zoo. There should be a list of enclosures (about a ten thousand of them) and it should be possible to display it in three ways:
all enclosures,
only enclosures that the currently logged in user has marked as favorite,
only enclosures that the currently logged in user has commented on.
In all of these cases the list could be too long to fit on a single page and therefore should be divided into multiple pages with a pagination bar.
In order to ease searching for a particular enclosure, all three modes should support additional filtering by a keyword (full-text search in enclosure names). I.e. the user should be able to e.g. display all enclosures marked as favorite that contain a given string in their names. Of course, the list can still be to large and pagination would be applicable here as well.
The question is - how to design the DAO layer to avoid code dupplication and spaghetti code full of conditions? Also, it would be fine to have the code divided into layers/areas of abstraction, so that e.g. the code for building the final SQL queries would not be scattered inconsistently across many different classes from different abstraction layers.
Assuming a traditional request/response web application style here is a sketch:
Represent the various filtering options as classes in supporting code for your DAO. Have the web client specify URL parameters representing the filtering options. You'll need a way to ensure that the filtering options are always sent in on each request, or store them on the user's session.
Map the filtering parameters to the filtering options and pass the options to your DAO. In your DAO's queries "expand" the filtering options into appropriate where claus(es) against the database.
For paging, have the concept of a paging "window". For example, you could have a class that represents the starting row and how many rows to return. Again, expand that class into a predicate executed against the database.
There are other ways to accomplish this (perhaps with one of the million frameworks that are around), but this is how I'd approach it if I had to develop it all from scratch.
Editing my original answer since I misread your criteria. Your DAO will be the same as any other basic DAO. It will (essentially) have a GET method for each of the three queries. If the user wants to narrow down the criteria after that, I would suggest using a jquery plugin like DataTables., assuming the amount of data that gets returned in the DAO methods isn't some outrageously huge amount. That plugin will allow you to add filters to each column that updates as you type, and also has sort, search, and paginate functionality.
I have found the Jquery datatables plug in extremely useful for simple, read only applications where I'd like to give the user pagination, sorting and searching of very large sets of data (millions of rows using server side processing).
I have a system for reusing this code but I end up doing the same thing over and over alot. I'd like to write a very generalized api that I essentially just need to configure the sql needed to retrieve the data used in the table. I am looking for a good design pattern/approach to do this. I've seen articles like this http://www.codeproject.com/Articles/359750/jQuery-DataTables-in-Java-Web-Applications and have a complete understanding of how server side processing works (have done it in java and asp.net many times). For someone to answer you will probably need to have a deep understanding of how server side processing works in java but here are some issues that come up with attempting to do this:
I generally run three separate queries. A count without the search clause, a count with the clause included, the query for the actual data. I haven't found an efficient way to do all 3 at once and doing so requires a lot of extra data to come back from db (ie counts over and over). The api needs to support behavior based on these three different queries and complex queries at that. I generally row number () over an index for the pagination to be relatively speedy with large data.
*where clause changes dynamically (user can search over a variable number of rows).
*order by clause changes for the same reason.
overall, each case is often pretty specific to the data we need. Is there a good way to abstract this so that I can do minimal work when I want to use the plug in server side.
So, the steps are as follows in most projects:
*extract the params the plug on sends to the server (alot of times my own are added, mostly date ranges)
*build the unfiltered count query (this is rarely dynamic).
*build the filtered count query (is dynamic)
*build the data query
*construct a model object of the table and return it as json.
A lot of the issues occur setting the prepared statements with a variable number of parameters. Dynamically generating the sql in a general way (say based on just column names) seems unlikely. I am wondering if someone else has created something they are using for this or if it sounds like a specific pattern is applicable. It has just occurred to me that creating a reusable filter may be helpful in java. Any advice would be greatly appreciated. Feel free to be language agnostic as the architecture is what I'm trying to figure out.
We have base search criteria where all request parameters relevant to DataTables are mapped onto class properties (fields) and custom search criteria class that extends base and contains specific to business logic fields for sutom search. Also on server side we have repository class that takes custom search criteria as an argument and makes queries to database.
If you are familiar with C#, you could check out custom binding code and example of usage.
You could do such custom binding in your Java code as well.
I want to read items from SQL table which are in Hierarchical manner. e.g. My table having Parent-Child relations.
Following is my table structure:
id name parentId
1 abc null
2 xyz 1
3 lmn 1
4 qwe null
5 asd 4
So I want to load all the Item hierarchically in one go, How can I do that?
I tried HashMap but it feels very complex....
thanks in advance...
You can retrieve the whole hierarchy through one SQL query, but the query depends on the database system - for instance, Oracle supports a "CONNECT BY" query for hierarchical queries. MySQL does not support this directly - but there are some very good approaches explained at Hierachical Queries in MySQL. I suggest that you try one of these. The basic approach is to create a function which imitates the CONNECT BY.
First of all, form your question it is not clear if you need help on SQL side or java data structure side.
I assume both i.e. you need to fetch data from database & then populate in appropriate java data structure.
So there is no direct way for this. Your data is mainly TREE STRUCTURE, so mainly you need to focus on java data structure for tree. So below are the steps that you need to take.
1) First create java class structure for tree structure. Refer below links for that.
Java tree data-structure?
http://www.java2s.com/Code/Java/Collections-Data-Structure/Yourowntreewithgenericuserobject.htm
http://www.codeproject.com/Articles/14799/Populating-a-TreeView-Control-from-the-Database
2) Then you don't need much from sql side if you finally need java objects. SO you can do plain select query to fetch data.
3) Now you need to have a helper method for preparing tree for you data using class structure form step 1. If you do better design, you tree class structure can be made self constructing using SQL data.