What are some examples of efficiently searching through a directory as you're typing a person's name?
Say for example, we have a database with 1 million users. We start typing in the search box: "sea", it will display every user's name on a scroll-able window that has "sea" on it (kind of like searching through a Skype directory). After changing a letter, the window should update immediately. All of this is coming from a SQL database. What are few efficient libraries, algorithms that can do this without much delay?
First consider changing the task from "name contains substring" to "name starts with substring". If this is possible, then add index on your name column in database table and use the query:
select name from table where name like :1 || '%'
Limit the number of returned rows using DBMS-specific syntax, for example, for Oracle add
and rownum < 20
This query should return your rows pretty fast.
If you really need "contains substring", then decide whether you want the search to be handled by database or by an external text indexing solution.
For database-contained solution you'll have to use a different approach depending on DBMS. Every one of these solutions requires configuration steps not described here.
For Oracle you can use Oracle Text, see
http://www.oracle.com/technetwork/documentation/index-098492.html
The query will look like
select name from table where contains(name, :1) > 0
For Postgres you can use Full Text Search.
You can also use a solution that is not dependent on the database, for example, see Apache Solr:
http://lucene.apache.org/solr/
for example
SELECT name
FROM Table
WHERE name LIKE '%sea%'
Related
In our application using a H2 database (version 1.4.196) we have a search on a varchar field doing either "contains" (column like '%searchterm%) or "begins with" (column like 'searchterm%) searches.
The table is quite large (approx. 400,000 entries) and the search turns out to be slow (varying between 3 seconds on my local development computer to 6 to 12 seconds on our customer's machines).
I found out that the column in question was not indexed and added an index. It turned out that the search time did not improve, even when I added an index hint explicitly to the query. explain revealed to me that no index was used in both cases.
From my experience with other database systems (e.g. MSSQL) I know that, at least for "begins with" queries, indexes can be used to improve search peformances.
As I did not find any related documentation for the H2 database my question:
Is it possible to use indexes in like queries in H2?
Most DB systems WILL use the index if you have a LIKE query... but only up to the first % in it. There is pretty much no DB system in existence where the mere act of having an index on the column would have any effect on the speed of, say: SELECT * FROM my_table WHERE my_column LIKE '%findme%';.
What you are presumably looking for is an index specifically designed to aid in searching. For example, the tsquery system that postgres has (see postgres tsquery documentation).
Another way to go is to add a dependency to your software that does this. The obvious choice there is Apache Lucene.
H2 has some support for this. Please refer to H2 documentation on full text search.
I am building one big application on Java. The problem I faced is that I have to search for a particular string in all database. I know in which column it should be but it could be in all tables.
Here it is:
I have 10 tables with the same columns and structure. One of them is storing ID of the product. I have search field in my Java application so the mySQL string behind this search needs to dig in all database (all ID columns of the tables).
Basically it has to look something like this:
Select *
from *.[database-name]
where [the particular column].*.[database] is lke %?%
I know code looks awful but it is only example to get the idea. I am sure that it is something about information_schema but I really cannot remember how and also cannot find answers as well(that satisfied me) in the site.
Thanks in advance.
Search Database with Keyword:
Is there a way to query the entire table of SQLite database for matching word. I am trying to place a word in keyword JTextField for it search the entire Job table and to return rows which contain that matching words. Each row representing a unique job.
I presume this below snipped structure would not achieve the result
SELECT * FROM Job WHERE Job MATCH 'Microsoft';
Any quick and simple recommendations?
It appears you can use the syntax you're suggesting as long as it's a full text search table:
http://www.phparch.com/2011/11/full-text-search-with-sqlite/
You should probably read the official SQLite documentation on FTS tables. Note that these are "virtual" tables and as such don't use the database file in the same way - this might not be what you want, but it matches the syntax you're looking for. The SQLite page linked has pretty good descriptions. Note that the FTS module is an optional extra which may not be installed.
You shouldn't need to worry about SQL injection attacks as long as you use placeholders. Look at this SO question, specifically the second answer with the code snippet. Essentially you put a ? in the statement, and then supply the string to substitute as a parameter to the query function. That should do SQL escaping for you.
i don't know sqlite but in mysql we have full text search which is used as
select * from table where match(name) against('value')
just see weather full text search is there or not in sqlite, i found this link verify it
http://www.sqlite.org/fts3.html#section_3
I am accessing a MySQL table that has over 1 million or more Records. I am using My SQL query browser which is unable to grab all the records and it break the connection in the middle.
Now I have to write a Java Program which access that particular table without being broken in the middle as this table will be modified and accessed frequently.
Can you experts suggest me how should do I go over this problem
either I create an Index on the table and how do I create index
There are different reasons why a MySQL connection might break during a query. Can you give the exact error message you receive?
A simplified explanation on how to add an index to the table for a simple query
Look at the field(s) in the WHERE
clause of the query
Add an index on the field(s) using
ALTER TABLE ADD INDEX
Use EXPLAIN on the query and check
if the query is actually using the
index.
IF you want more specific help, Post the SHOW CREATE TABLE and the EXPLAIN of your query.
MySQL query browser limits the number of records to be displayed for performance reasons, because it is an interactive program and nobody like to wait for half an hour before the program crashes with an out-of-memory error. You can change these limits in the settings.
Your Java program will face similar problems.
When using large datasets it is important to plan how you are going to access that dataset and create the necessary indexes.
It would be useful to edit the question to show the structure of the data. Generqlly it looks like this :
CREATE INDEX idx_customer_name ON customer (name);
Here are more details
If you just want to dump the data to work on the data using Excel you can try this on the commandline
mysqldump -u [username] -p -t -T/path/to/directory [database] --fields-enclosed-by=\" --fields-terminated-by=,
In my experience this is a very painful exercise as Excel really is not made to deal with this amount of rows, and the dump format usually is slightly, but infuriatingly incompatible.
Your best bet is to invest an hour of your time to go through a SQL tutorial like sql fundamentals and play with MySQL query browser to get a feel of what you can do with SQL. I guarantee your investment paid itself back by tomorrow.
I am not very well used to MySQL programming, but generally indexes are used to arrange the values of one or more columns in a database table in specific order.
SYNTAX
CREATE INDEX IndexName ON tableName (column);
Just go through this tutorial for more information,
http://dev.mysql.com/doc/refman/5.0/en/create-index.html
I want to display a list of all the users in my site but I only want to display 10 people per age. I don't know how exactly to do this. I know how to do it by just displaying all the users in one page but that's not very good is it?
If I do it with the code I have now, it will only get the first ten users over and over again.
I want to be able to get all the users for a one time query, store it globally and then move through the list retrieving the next 10 and so on for display.
I am developing on appengine using Java and the Spring Framework some of the solutions I have been thinking about,
Store in the session and go through the list (very bad I guess)
hand it to the JSP, specifically to one of the scopes, page, request etc. But I think request will not work.
Look for a Spring controller that can handle this.
Generally speaking, you would use a form variable on your page (via GET or POST) called 'page', which would be a number. When you receive that in the servlet you would calculate a range based on the page number and configured rows per page.
Take a look at Paging through large datasets (yes it's Python but the same principles apply) and Queries and Indexes from the Google App Engine documentation.
Take a look at http://valuelist.sourceforge.net/
If you keep page size at 10 then you can retrieve your 10 users per age group for each page based on page number:
SELECT TOP 10 users FROM myusers
WHERE AGE = function(page_number)
ORDER BY some_ordering
I hope that JPA + appengine support such type of query.
Some database engines provide handy extensions to SQL for just this purpose. Like, in MySQL you can say something like "select ... whatever ... limit 50,10", where "50" is the row to start with and 10 is the number of rows to retrieve. Then on your display page you simply put next and previous buttons that pass the appropriate starting row number back to the server for the next run at the query.
If the SQL engine you're using has no such handy function, then you have to build an query-specific "where" clause based on the sort order.
To take a simple case, suppose in your example you are displaying the records in order by "user_name". You can use Statement.setMaxRows(10) to limit any queries to 10 rows. Then on your first call you execute, say, "select ... whatever ... from user order by user_name". Save the last user_name found. In your next button, you pass this user_name back to the server, and the query for the next call is "select ... whatever ... from user where user_name>'xxx' order by user_name", where 'xxx' is the last user_name from the previous call. Do the setMaxRows again so you are again limited to 10 rows of output. You can then let the user step through the entire output this way.
Letting the user go backwards is a bit of a pain. I've done it by keeping a table in a session variable with the starting key value for each page.