So I have a record with following field :
"fullName" : "Virat Kohli"
I have written the following multi_match query that should fetch this record :
GET _search
{
"query": {
"multi_match": {
"query": "*kohli*",
"fields": [
"fullName^1.0",
"team^1.0"
],
"type": "phrase_prefix",
"operator": "OR",
"slop": 0,
"prefix_length": 0,
"max_expansions": 50,
"zero_terms_query": "NONE",
"auto_generate_synonyms_phrase_query": true,
"fuzzy_transpositions": true,
"boost": 1
}
}
}
This works fine.
But when I remove the letter 'k' from query and change it to :
"query": "*ohli*"
It doesn't fetch any record.
Any reason why this is happening? How can I modify the query to get the record returned with the above modification?
first let me explain you why your existing query didn't work and then the solution of it.
Problem : you are using the multi_match query with type phrase_prefix and as explained in the documentation it makes a prefix query on the last search term and in your case you have only 1 search term so on that Elasticsearch will perform the phrase query.
And prefix query works on the exact tokens, and you are using standard analyzer mostly, default for text fields so for fullName field it will have virat and kohli and your search term also generates kohli(notice smallcase k) as standard analyzer also lowercase the tokens, above you can check with the explain API output in your first request as shown below.
"_explanation": {
"value": 0.2876821,
"description": "max of:",
"details": [
{
"value": 0.2876821,
"description": "weight(fullName:kohli in 0) [PerFieldSimilarity], result of:",
"details": [
{
(note he search term in the weight)
Solution
As you are trying to use the wildcard in your query, best solution is to use the wildcard query against your field as shown below to get results in both case.
{
"query": {
"wildcard": {
"fullName": {
"value": "*ohli",
"boost": 1.0,
"rewrite": "constant_score"
}
}
}
}
And SR
"hits": [
{
"_shard": "[match_query][0]",
"_node": "BKVyHFTiSCeq4zzD-ZqMbA",
"_index": "match_query",
"_type": "_doc",
"_id": "1",
"_score": 1.0,
"_source": {
"id": 2,
"fullName": "Virat Kohli",
"team": [
"Royal Challengers Bangalore",
"India"
]
},
"_explanation": {
"value": 1.0,
"description": "fullName:*ohli",
"details": []
}
}
]
I am working with elastic search and am trying to look for a substring inside a field. For example - searching for the string tac in stack overflow . I am using the MultiMatchQuery for this but it does not work. Here is a snippet of my code (first_name is the field name).
searchString = "*" + searchString.toLowerCase() + "*";
MultiMatchQueryBuilder mqb = new MultiMatchQueryBuilder("irs", first_name);
mqb.type(MultiMatchQueryBuilder.Type.PHRASE);
BoolQueryBuilder searchQuery = boolQuery();
searchQuery.should(mqb);
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
queryBuilder.withQuery(searchQuery);
NativeSearchQuery query = queryBuilder.build();
When I search for tac it does not return any results. When I search for stack or overflow it does return stack overflow.
So it looks for the exact string. I tried using MultiMatchQueryBuilder.Type.PHRASE_PREFIX but it looks for the phrases starting with the substring. It works with strings like stac or overf but not tac or tack.
Any suggestions on how to fix it?
Macth query is analyzed and applied the same analyzer which is applied during the index time, I believe you are using the standard analyzer, which generated below tokens
POST http://localhost:9200/_analyze
{
"text": "stack overflow",
"analyzer" : "standard"
}
{
"tokens": [
{
"token": "stack",
"start_offset": 0,
"end_offset": 5,
"type": "<ALPHANUM>",
"position": 0
},
{
"token": "overflow",
"start_offset": 6,
"end_offset": 14,
"type": "<ALPHANUM>",
"position": 1
}
]
}
Hence searching for tac doesn't match any token in an index, you need to change the analyzer so that it matches the query time tokens to index time tokens.
n-gram tokenizer can solve the issue.
Example
Index mapping
{
"settings": {
"analysis": {
"filter": {
"autocomplete_filter": {
"type": "ngram",
"min_gram": 1,
"max_gram": 10
}
},
"analyzer": {
"autocomplete": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"autocomplete_filter"
]
}
}
},
"index.max_ngram_diff" : 10
},
"mappings": {
"properties": {
"title": {
"type": "text",
"analyzer": "autocomplete",
"search_analyzer": "standard"
}
}
}
}
Index sample doc
{
"title" : "stack overflow"
}
And search query
{
"query": {
"match": {
"title": "tac"
}
}
}
And search result
"hits": [
{
"_index": "65241835",
"_type": "_doc",
"_id": "1",
"_score": 0.4739784,
"_source": {
"title": "stack overflow"
}
}
]
}
I am trying to update a value in my index via java api using UpdateRequest which accepts three arguments.
Index
document
id
Question - I know what my index name is but i am not sure what value should be passed in document and id field.
SAMPLE DATA
{
"took": 2,
"timed_out": false, "_shards": { "total": 5, "successful": 5, "failed": 0 },
"hits": {
"total": 1,
"max_score": 0.94064164,
"hits": [
{
"_index": "ticketdump",
"_type": "event",
"_id": "AVefK2vFmf0chKzzBkzy",
"_score": 0.94064164,
"_source": {
"clientversion": "123465",
"queue": "test,test",
"vertical": "test",
"troubleshooting": "test",
"reason": "test",
"status": "test",
"ticketversion": "1132465",
"apuid": 1,
"golive": "2014-07-14",
"clientname": "test",
"message": "test",
"product": "test",
"clientid": 1,
"createddatetime": "2016-05-03 09:43:48",
"area": "test",
"developmentfix": "test",
"actiontaken": "test",
"categoryname": "test",
"parentcategory": "test",
"problemdef": "test",
"ticketid": 1
}
}
]
}
}
I tried to pass _source object but it gave document missing error.Maybe I am missing the concept?
JAVA CODE
UpdateRequest updateRequest = new UpdateRequest(
"ticketdump",
js.getJSONObject("hits")
.getJSONArray("hits")
.getJSONObject(0)
.getJSONObject("_source")
.toString(),
"1"
).script(new Script("ctx._source.message = \"bhavik\""));
client.update(updateRequest).get();
Actually, your UpdateRequest accepts 3 parameters
Index
Type
Id
By the following data you can see that:
Index = ticketdump
Type = event
Id = AVefK2vFmf0chKzzBkzy
I have 999 documents which I am using for experimenting with elastic search.
There is a field f4 in my type mapping which is analyzed and has following settings for analyzer :
"myNGramAnalyzer" => [
"type" => "custom",
"char_filter" => ["html_strip"],
"tokenizer" => "standard",
"filter" => ["lowercase","standard","asciifolding","stop","snowball","ngram_filter"]
]
My filter is as below :
"filter" => [
"ngram_filter" => [
"type" => "edgeNGram",
"min_gram" => "2",
"max_gram" => "20"
]
]
I have value for field f4 as "Proj1", "Proj2", "Proj3"...... so on.
Now when I try to do search using cross fields for "proj1" string, I was expecting document with "Proj1" to be returned at the top of the response with max score. But it doesn't. Rest all the data is almost same in content.
Also I don't understand why it matches all 999 document?
Following is my search :
{
"index": "myindex",
"type": "mytype",
"body": {
"query": {
"multi_match": {
"query": "proj1",
"type": "cross_fields",
"operator": "and",
"fields": "f*"
}
},
"filter": {
"term": {
"deleted": "0"
}
}
}
}
My search response is :
{
"took": 12,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 999,
"max_score": 1,
"hits": [{
"_index": "myindex",
"_type": "mytype",
"_id": "42",
"_score": 1,
"_source": {
"f1": "396","f2": "125650","f3": "BH.1511AI.001",
"f4": "Proj42",
"f5": "BH.1511AI.001","f6": "","f7": "","f8": "","f9": "","f10": "","f11": "","f12": "","f13": "","f14": "","f15": "","f16": "09/05/16 | 01:02PM | User","deleted": "0"
}
}, {
"_index": "myindex",
"_type": "mytype",
"_id": "47",
"_score": 1,
"_source": {
"f1": "396","f2": "137946","f3": "BH.152096.001",
"f4": "Proj47",
"f5": "BH.1511AI.001","f6": "","f7": "","f8": "","f9": "","f10": "","f11": "","f12": "","f13": "","f14": "","f15": "","f16": "09/05/16 | 01:02PM | User","deleted": "0"
}
},
//.......
//.......
//MANY RECORDS IN BETWEEN HERE
//.......
//.......
{
"_index": myindex,
"_type": "mytype",
"_id": "1",
"_score": 1,
"_source": {
"f1": "396","f2": "142095","f3": "BH.705215.001",
"f4": "Proj1",
"f5": "BH.1511AI.001","f6": "","f7": "","f8": "","f9": "","f10": "","f11": "","f12": "","f13": "","f14": "","f15": "","f16": "09/05/16 | 01:02PM | User","deleted": "0"
}
//.......
//.......
//MANY RECORDS IN BETWEEN HERE
//.......
//.......
}]
}
}
Any thing that I am doing wrong or missing? (Apologies for lengthy question, but I thought to give all possible information discarding unnecessary other code).
EDITED :
Term vector response
{
"_index": "myindex",
"_type": "mytype",
"_id": "10",
"_version": 1,
"found": true,
"took": 9,
"term_vectors": {
"f4": {
"field_statistics": {
"sum_doc_freq": 5886,
"doc_count": 999,
"sum_ttf": 5886
},
"terms": {
"pr": {
"doc_freq": 999,
"ttf": 999,
"term_freq": 1,
"tokens": [{
"position": 0,
"start_offset": 0,
"end_offset": 6
}]
},
"pro": {
"doc_freq": 999,
"ttf": 999,
"term_freq": 1,
"tokens": [{
"position": 0,
"start_offset": 0,
"end_offset": 6
}]
},
"proj": {
"doc_freq": 999,
"ttf": 999,
"term_freq": 1,
"tokens": [{
"position": 0,
"start_offset": 0,
"end_offset": 6
}]
},
"proj1": {
"doc_freq": 111,
"ttf": 111,
"term_freq": 1,
"tokens": [{
"position": 0,
"start_offset": 0,
"end_offset": 6
}]
},
"proj10": {
"doc_freq": 11,
"ttf": 11,
"term_freq": 1,
"tokens": [{
"position": 0,
"start_offset": 0,
"end_offset": 6
}]
}
}
}
}
}
EDITED 2
Mappings for field f4
"f4" : {
"type" : "string",
"index_analyzer" : "myNGramAnalyzer",
"search_analyzer" : "standard"
}
I have updated to use standard analyzer for query time, which has improved the results but still not what I expected.
Instead of 999 (all documents) now it return 111 documents like "Proj1", "Proj11", "Proj111"......"Proj1", "Proj181"......... etc.
Still "Proj1" is in between the results and not at the top.
There is no index_analyzer (at least not from Elasticsearch version 1.7). For mapping parameters you can use analyzer and search_analyzer.
Try the following steps in order to make it work.
Create myindex with analyzer settings:
PUT /myindex
{
"settings": {
"analysis": {
"filter": {
"ngram_filter": {
"type": "edge_ngram",
"min_gram": 2,
"max_gram": 20
}
},
"analyzer": {
"myNGramAnalyzer": {
"type": "custom",
"tokenizer": "standard",
"char_filter": "html_strip",
"filter": [
"lowercase",
"standard",
"asciifolding",
"stop",
"snowball",
"ngram_filter"
]
}
}
}
}
}
Add mappings to mytype (to make it short I just mapped the relevant fields):
PUT /myindex/_mapping/mytype
{
"properties": {
"f1": {
"type": "string"
},
"f4": {
"type": "string",
"analyzer": "myNGramAnalyzer",
"search_analyzer": "standard"
},
"deleted": {
"type": "string"
}
}
}
Index some data:
PUT myindex/mytype/1
{
"f1":"396",
"f4":"Proj12" ,
"deleted": "0"
}
PUT myindex/mytype/2
{
"f1":"42",
"f4":"Proj22" ,
"deleted": "1"
}
Now try your query:
GET myindex/mytype/_search
{
"query": {
"multi_match": {
"query": "proj1",
"type": "cross_fields",
"operator": "and",
"fields": "f*"
}
},
"filter": {
"term": {
"deleted": "0"
}
}
}
It should return document #1. It worked for me with Sense. I am using Elasticsearch 2.X versions.
Hope I have managed to help :)
After hours of spending time to find a solution to this, I finally made it work.
So I kept everything same as mentioned in my question, using n gram analzyer while indexing data. The only thing I had to change was, to use the all field in my search query as a bool query with my existing multi-match query.
Now my result for search text Proj1 would return me results in an order such as Proj1, Proj121, Proj11, etc.
Although this does not return the exact order like Proj1, Proj11, Proj121, etc, but still it closely resembles the result that I wanted.
I have term in my dataset which does not give any search results but is present in a document.
If I request a term vector:
GET index_5589b14f3004fb6be70e4724/document_set/382.txt/_termvector
{
"fields" : ["plain_text", "pdf_text"],
"term_statistics" : true,
"field_statistics" : true
}
The term vector has this word:
...
"advis": { //porter stemmed version of the word "advising"
"doc_freq": 1,
"ttf": 1,
"term_freq": 1,
"tokens": [
{
"position": 81,
"start_offset": 412,
"end_offset": 420
}
]
},...
"air": {
But when I search this word to retrieve all the documents where it has occurred I get zero hits:
GET index_5589b14f3004fb6be70e4724/document_set/_search
{
"query": {
"multi_match": {
"query": "advis",
"fields": ["plain_text", "pdf_text"]
}
},
"explain": true
}
Why is this happening?
This is due to the fact that the search term is getting analyzed most probably in the above example advis is being stemmed to advi.
You can explicitly specify keyword analyzer in the query and you should get the values
GET index_5589b14f3004fb6be70e4724/document_set/_search
{
"query": {
"multi_match": {
"query": "advis",
"fields": ["plain_text", "pdf_text"],
"analyzer" : "keyword"
}
},
"explain": true
}