I am using SQLite FTS extension in my iOS application.
It performs well but the problem is that it matches only string prefixes (or starts with keyword search).
i.e.
This works:
SELECT FROM tablename WHERE columnname MATCH 'searchterm*'
but following two don't:
SELECT FROM tablename WHERE columnname MATCH '*searchterm'
SELECT FROM tablename WHERE columnname MATCH '\*searchterm\*'
Is there any workaround for this or any way to use FTS to build a query similar to LIKE '%searchterm%' query.
EDIT:
As pointed out by Retterdesdialogs, storing the entire text in reverse order and running a prefix search on a reverse string is a possible solution for ends with/suffix search problem, which was my original question, but it won't work for 'contains' search. I have updated the question accordingly.
In my iOS and Android applications, I have shied away from FTS search for exactly the reason that it doesn't support substring matches due to lack of suffix queries.
The workarounds seem complicated.
I have resorted to using LIKE queries, which while being less performant than MATCH, served my needs.
The workaround is to store the reverse string in an extra column. See this link (its not exactly the same it should give a idea):
Search Suffix using Full Text Search
To get it to work for contains queries, you need to store all suffixes of the terms you want to be able to search. This has the downside of making the database really large, but that can be avoided by compressing the data.
SQLite FTS contains and suffix matches
Related
What are some examples of efficiently searching through a directory as you're typing a person's name?
Say for example, we have a database with 1 million users. We start typing in the search box: "sea", it will display every user's name on a scroll-able window that has "sea" on it (kind of like searching through a Skype directory). After changing a letter, the window should update immediately. All of this is coming from a SQL database. What are few efficient libraries, algorithms that can do this without much delay?
First consider changing the task from "name contains substring" to "name starts with substring". If this is possible, then add index on your name column in database table and use the query:
select name from table where name like :1 || '%'
Limit the number of returned rows using DBMS-specific syntax, for example, for Oracle add
and rownum < 20
This query should return your rows pretty fast.
If you really need "contains substring", then decide whether you want the search to be handled by database or by an external text indexing solution.
For database-contained solution you'll have to use a different approach depending on DBMS. Every one of these solutions requires configuration steps not described here.
For Oracle you can use Oracle Text, see
http://www.oracle.com/technetwork/documentation/index-098492.html
The query will look like
select name from table where contains(name, :1) > 0
For Postgres you can use Full Text Search.
You can also use a solution that is not dependent on the database, for example, see Apache Solr:
http://lucene.apache.org/solr/
for example
SELECT name
FROM Table
WHERE name LIKE '%sea%'
Im using google app engine data-store built in eclipse using my model for the table. The id is just the date and time from android.
I can query by a row like this and it does work!
select from Quotes as Quotes ORDER BY votes DESC
I want to get my results back by my entities id however this query does not work
select from Quotes as Quotes ORDER BY Id DESC
Here is my table. How can I query by my id/Name and trust me ive tried
select from Quotes as Quotes ORDER BY ID/Name DESC
edit: you probably notice i have a dummyid. I do not want to use that row because I made it in a very hacky way and requires extra loading on the users side.
Oh, dear. I see the problem, now. You have a column named ID/Name. It's usually wise to keep identifiers limited to alphanumeric characters.
Can you rename the column? That would be the best step forward.
If that's not an option, you can wrap it in backticks so that it's treated as an identifier:
SELECT * FROM Quotes ORDER BY `ID/Name` DESC;
See SQL Fiddle, which almost certainly won't match your schema but should get the point across.
That Id/Name is the key field, imagine it is similar to primary key. to refer to that field in query, use
__key__
Example: select * from EntityTable where __key__ = Key('EntityTable', ....)
In your example, using date/time as key name is not really helpful, maybe you can find another info to be used as key.
Now I have a situation where I need to make some comparisons and result filtration that is not very simple to do, what I want is something like Lucenes search but only I will develop it, it is not my decision though I would have gone with Lucene.
What I will do is:
Find the element according to full word match of a certain field, if not then check if it starts with it the check if it just contains.
Every field has its weight according to matching case(full->begins->contains) and its priority to me.
After one has matched I will also check the weight of the other fields as well to make a final total row weight.
Then I will return an Map of both rows and their weights.
Now I realized that this is not easy done by hibernate's HQL meaning I would have to run multiple queries to do this.
So my question is should I do it in java meaning should I retrieve all records and do my calculations to find my target, or should I do it in hibernate by executing multiple queries? which is better according to performance and speed ?
Unfortunately, I think the right answer is "it depends": how many words, what data structure, whether the data fits in memory, how often you have to do the search, etc.
I am inclined to think that a database is a better solution, even if Hibernate is not part of it. You might need to learn how to write better SQL. Perhaps the dynamic SQL that Hibernate generates for you isn't sufficient. Proper JOINs and indexing might make this perform nicely.
There might be a third way to consider: Lucene and indexing. I'd need to know more about your problem to decide.
I am working with the Lucene and Derby databases. Lucene contains the text index, and Derby has information regarding additional user data. For example, each document has a tag. For this purpose the Derby database has two tables
TAGS:
ID
Name
LUCENETAGS:
ID
LUCENEID (docID in Lucene, not a field)
TAGID
I want a user to be able to search something like:
very interesting text AND tag:fun
Changing the structure in a way that tag is a Lucene field is not an option.
Thank you!
I believe you'll have to simply perform your text search in Lucene, and then filter your results based on the result of a query into a Derby.
If few documents will match a particular tag, you could also query the database for the IDs to be queried, and rewrite the query like:
(very interesting text) AND id:(1 2 3 etc.)
Probably not feasible, but in the case that tags are pretty sparse, it might be worth considering.
I do wonder, though, why a field can't be added to the index, duplicating the stored value in the Derby Database. In any implementation you choose to get what you want from your stated structure, you will see much poorer performance, and more complexity for you to deal with, than if the data were available in the index as well.
Search Database with Keyword:
Is there a way to query the entire table of SQLite database for matching word. I am trying to place a word in keyword JTextField for it search the entire Job table and to return rows which contain that matching words. Each row representing a unique job.
I presume this below snipped structure would not achieve the result
SELECT * FROM Job WHERE Job MATCH 'Microsoft';
Any quick and simple recommendations?
It appears you can use the syntax you're suggesting as long as it's a full text search table:
http://www.phparch.com/2011/11/full-text-search-with-sqlite/
You should probably read the official SQLite documentation on FTS tables. Note that these are "virtual" tables and as such don't use the database file in the same way - this might not be what you want, but it matches the syntax you're looking for. The SQLite page linked has pretty good descriptions. Note that the FTS module is an optional extra which may not be installed.
You shouldn't need to worry about SQL injection attacks as long as you use placeholders. Look at this SO question, specifically the second answer with the code snippet. Essentially you put a ? in the statement, and then supply the string to substitute as a parameter to the query function. That should do SQL escaping for you.
i don't know sqlite but in mysql we have full text search which is used as
select * from table where match(name) against('value')
just see weather full text search is there or not in sqlite, i found this link verify it
http://www.sqlite.org/fts3.html#section_3