i am trying to achieve below condition
orgId = "z2store" and type = "web" and dateTime = "12:17:08"
below query i have written
GET /sample/_search
{
"bool" : {
"must" : [
{
"term" : {
"orgId" : {
"value" : "z2store",
"boost" : 1.0
}
}
},
{
"term" : {
"type" : {
"value" : "web",
"boost" : 1.0
}
}
},
{
"query_string" : {
"query" : "12:17:08",
"default_field" : "dateTime",
"fields" : [ ],
"type" : "best_fields",
"default_operator" : "or",
"max_determinized_states" : 10000,
"enable_position_increments" : true,
"fuzziness" : "AUTO",
"fuzzy_prefix_length" : 0,
"fuzzy_max_expansions" : 50,
"phrase_slop" : 0,
"escape" : false,
"auto_generate_synonyms_phrase_query" : true,
"fuzzy_transpositions" : true,
"boost" : 1.0
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
below is my java code
BoolQueryBuilder boolQuery = new BoolQueryBuilder().must(QueryBuilders.termQuery("orgId", orgId))
.must(QueryBuilders.termQuery("type", "web"));
QueryStringQueryBuilder builder = new QueryStringQueryBuilder("12:17:08");
builder.defaultField("dateTime").queryString();
boolQuery.must(builder);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder().query(builder)
.from((batchNumber - 1) * batchSize).size(batchSize)
.sort("#timestamp", SortOrder.DESC);
Above query is not working. Any help will be appreciated. I am using elasticSearch 7.4.
You can create your dateTime field with type as date and giving format as hour_minute_second(which takes format as HH:mm:ss) . You can read more about different date formats here https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html.
Below is the mapping of dateTime field:
{
"mappings": {
"properties": {
"dateTime": {
"type" : "date",
"format" : "hour_minute_second"
}
}
}
}
Now when you search data with below search query :
{
"query" : {
"bool" : {
"must" : [
{
"term" : {
"orgId" : {
"value" : "z2store",
"boost" : 1.0
}
}
},
{
"term" : {
"type" : {
"value" : "web",
"boost" : 1.0
}
}
},
{
"term" :{
"dateTime":"12:17:08"
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
}
You will get your required result :
"hits": [
{
"_index": "datetimeindexf",
"_type": "_doc",
"_id": "1",
"_score": 1.5753641,
"_source": {
"dateTime": "12:17:08",
"orgId": "z2store",
"type": "web"
}
}
]
Related
There is this opensearch query constructed using openserch-java
GET eventsearch/_search
{
"aggregations": {
"WEB": {
"aggregations": {
"eventDate": {
"date_histogram": {
"extended_bounds": {
"max": "2022-12-01T00:00:00Z",
"min": "2022-01-01T00:00:00Z"
},
"field": "eventDate",
"fixed_interval": "1d",
"min_doc_count": 0
}
}
},
"filter": {
"term": {
"channel": {
"value": "WEB",
"case_insensitive": true
}
}
}
}
},
"query": {
"bool": {
"filter": [
{
"range": {
"eventDate": {
"from": "2022-01-01T00:00:00Z",
"to": "2022-12-01T00:00:00Z"
}
}
}
],
"must": [
{
"match_all": {}
}
]
}
},
"size": 0
}
Running query, the response is this:
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 26,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"WEB" : {
"doc_count" : 25,
"eventDate" : {
"buckets" : [
{
"key_as_string" : "2022-01-01T00:00:00.000Z",
"key" : 1640995200000,
"doc_count" : 0
},
{
"key_as_string" : "2022-01-02T00:00:00.000Z",
"key" : 1641081600000,
"doc_count" : 0
},
{
"key_as_string" : "2022-01-03T00:00:00.000Z",
"key" : 1641168000000,
"doc_count" : 0
},
{
"key_as_string" : "2022-01-04T00:00:00.000Z",
"key" : 1641254400000,
"doc_count" : 0
},
....................
]
}
}
}
}
In java I need to perform this query and get the results from there.
But after using the opensearchclient.search and then get the "aggregations" list method, I receive this (image attached) and get
If I try to get the "WEB" from the Map, there is no other "eventDate" aggregation to fetch.
Is there a way to fetch this inner aggregation using opensearch-java client? I had no luck with documentation.
opensearch-java 2.1.0
There is currently no feature like this, it exists an open bug, with merged code, but not released.
https://github.com/opensearch-project/opensearch-java/issues/197
Mongodb 4.2.15
I'm tring to use mongo Updates with Aggregation Pipeline
My request is very big but here it's core structure
db.runCommand({
"update": "collectionName",
"updates": [
{
"q": { ... },
"u": [
{
"$project": { ... },
"$set": { ... },
"$set": { ... },
"$project": { ... }
}
],
"multi": false,
"upsert": true
}
]
});
After the first execute I receive a result with newly created object's _id
{
"n" : 1,
"nModified" : 0,
"upserted" : [
{
"index" : 0,
"_id" : ObjectId("619997f11501d6eb40c6f64a")
}
],
"opTime" : {
"ts" : Timestamp(1637455857, 61),
"t" : NumberLong(5)
},
"electionId" : ObjectId("7fffffff0000000000000005"),
"ok" : 1.0,
"$clusterTime" : {
"clusterTime" : Timestamp(1637455857, 61),
"signature" : {
"hash" : { "$binary" : "hA0tf5DXMqTNmnVXdMVnpnAKCU0=", "$type" : "00" },
"keyId" : NumberLong(7018546168816730116)
}
},
"operationTime" : Timestamp(1637455857, 61)
}
After the second execution of the same request there is no modified object's _id
{
"n" : 1,
"nModified" : 1,
"opTime" : {
"ts" : Timestamp(1637456057, 19),
"t" : NumberLong(5)
},
"electionId" : ObjectId("7fffffff0000000000000005"),
"ok" : 1.0,
"$clusterTime" : {
"clusterTime" : Timestamp(1637456057, 19),
"signature" : {
"hash" : { "$binary" : "U2yCP6nXUrjBN9ZiLanyl0rgxww=", "$type" : "00" },
"keyId" : NumberLong(7018546168816730116)
}
},
"operationTime" : Timestamp(1637456057, 19)
}
The thing is that my filter conditions do not contain _id of the object but I need to return it with response. I see no useful request configurations. Any suggestions is it possible to get _id at response on update case?
This is my sample es index document:
"hits" : [
{
"_index" : "project_note",
"_type" : "project_note",
"_id" : "19",
"_score" : 1.0,
"_source" : {
"createTime" : "2021-10-04T13:43:55.330",
"createTimeInMs" : 1633333435330,
"createdBy" : "test",
"editTime" : "2021-10-04T13:43:55.330",
"editTimeInMs" : 1633333435330,
"editedBy" : "test",
"versionId" : 1,
"id" : "19",
"organizationId" : "28",
"accessLevel" : "PUBLIC",
"status" : "ACTIVE",
"projectId" : "95",
"userId" : 129,
"noteType" : "SYSTEM_GENERATED",
"projectDemographicLogId" : "1"
},
{
"_index" : "project_note",
"_type" : "project_note",
"_id" : "19",
"_score" : 1.0,
"_source" : {
"createTime" : "2021-10-04T13:43:55.330",
"createTimeInMs" : 1633333435330,
"createdBy" : "test",
"editTime" : "2021-10-04T13:43:55.330",
"editTimeInMs" : 1633333435330,
"editedBy" : "test",
"versionId" : 1,
"id" : "19",
"organizationId" : "28",
"accessLevel" : "PUBLIC",
"status" : "ACTIVE",
"projectId" : "95",
"userId" : 129
}
]
In the first doc, it has noteType but in the second, I don't have that field stored in db.
I want to exclude the documents where noteType==null or noteType is absent.
But, I am getting only the docs which have noteType="SYSTEM_GENERATED"
My approach:
{
"query":
{
"bool" : {
"must" : [
{
"term" : {
"projectId" : {
"value" : "95",
"boost" : 1.0
}
}
},
{
"range" : {
"createTimeInMs" : {
"from" : null,
"to" : 1633594455000,
"include_lower" : true,
"include_upper" : true,
"boost" : 1.0
}
}
}
],
"must_not" : [
{
"term" : {
"noteType" : {
"value" : "SYSTEM_GENERATED",
"boost" : 1.0
}
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
}
Equivalent java code:
BoolQueryBuilder queryBuilder= QueryBuilders.boolQuery();
queryBuilder.must(QueryBuilders.termQuery("projectId", requestInfo.getProjectId()));
queryBuilder.must(rangeQuery("createTimeInMs").lte(requestInfo.getCreateTimeInMs()));
if(!requestInfo.isIncludeLog()) {
queryBuilder.mustNot(QueryBuilders.termQuery("noteType", Defs.SYSTEM_NOTE_TYPE));
}
If only the must_not part of the query is used (excluding the must part)
{
"query": {
"bool": {
"must_not": [
{
"term": {
"noteType.keyword": {
"value": "SYSTEM_GENERATED",
"boost": 1.0
}
}
}
],
"adjust_pure_negative": true,
"boost": 1.0
}
}
}
The search result is similar to what you expect to get
"hits": [
{
"_index": "69477995",
"_type": "_doc",
"_id": "2",
"_score": 0.0,
"_source": {
"createTime": "2021-09-26T15:54:08.373",
"createTimeInMs": 1632650048373,
"createdBy": "test",
"editTime": "2021-09-26T15:54:08.373",
"editTimeInMs": 1632650048373,
"editedBy": "test",
"versionId": 1,
"id": "18",
"note": "note-1, simple note ",
"organizationId": "28",
"accessLevel": "PUBLIC",
"status": "ACTIVE",
"taskId": "5",
"userId": 129
}
}
]
I am trying to generate a query using the Java RestHighLevelClient of Elasticsearch similar to this:
GET /field_search/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"ENTRY_ID": "ttttt"
}
},
{
"match": {
"MODULE_ID": "xxxxx"
}
},
{
"match": {
"COMPANY_ID": "22244"
}
},
{
"match": {
"DELETED": false
}
}
]
}
}
}
This is my code that I am using to generate it
BoolQueryBuilder boolQueryBuilder1 = new BoolQueryBuilder();
boolQueryBuilder1.must().add(QueryBuilders.matchQuery("MODULE_ID", moduleId));
boolQueryBuilder1.must().add(QueryBuilders.matchQuery("COMPANY_ID", companyId));
........
I have skipped some part of it to keep it small. But i use a BoolQueryBuilder and the query it generates is something like this:
{
"query": {
"bool" : {
"must" : [
{
"match" : {
"MODULE_ID" : {
"query" : "xxxxx",
"operator" : "OR",
"prefix_length" : 0,
"max_expansions" : 50,
"fuzzy_transpositions" : false,
"lenient" : false,
"zero_terms_query" : "NONE",
"auto_generate_synonyms_phrase_query" : false,
"boost" : 1.0
}
}
},
{
"match" : {
"COMPANY_ID" : {
"query" : "22244",
"operator" : "OR",
"prefix_length" : 0,
"max_expansions" : 50,
"fuzzy_transpositions" : false,
"lenient" : false,
"zero_terms_query" : "NONE",
"auto_generate_synonyms_phrase_query" : false,
"boost" : 1.0
}
}
},
{
"match" : {
"DELETED" : {
"query" : false,
"operator" : "OR",
"prefix_length" : 0,
"max_expansions" : 50,
"fuzzy_transpositions" : false,
"lenient" : false,
"zero_terms_query" : "NONE",
"auto_generate_synonyms_phrase_query" : false,
"boost" : 1.0
}
}
},
{
"match" : {
"ENTRY_ID" : {
"query" : ttttt,
"operator" : "OR",
"prefix_length" : 0,
"max_expansions" : 50,
"fuzzy_transpositions" : false,
"lenient" : false,
"zero_terms_query" : "NONE",
"auto_generate_synonyms_phrase_query" : false,
"boost" : 1.0
}
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
}
It adds additional things into the query. Using the above normal query my results come back correctly but with the java generated query my results are none so how can i build the same query using Java client?
"adjust_pure_negative" : true
This is your problem, set it to false or delete it.
Read here why this happens.
Elasticsearch Java client's QueryBuider instance is adding properties like
1) order
2) min_doc_count
3) shard_min_doc_count
4) show_term_doc_count_error
5) lang
6) gap_policy
to the final Json query. My query works as expected without those properties. I want to prevent those properties from being added to my final query.
Java:
FilterAggregationBuilder aggregation = AggregationBuilders.filter("id", QueryBuilders.termsQuery("id",
"my-name"));
TermsAggregationBuilder lev1Agg = AggregationBuilders.terms("id").field("id");
lev1Agg.size(1);
lev1Agg.subAggregation(AggregationBuilders.sum("familyMemberCount").field("membersInFamily"));
lev1Agg.subAggregation(AggregationBuilders.sum("totalKidsInFamily").field("kidsInFamily"));
Map<String, String> bucketsPathsMap = new HashMap<>();
bucketsPathsMap.put("familyMemberCount", "familyMemberCount");
bucketsPathsMap.put("totalKidsInFamily", "totalKidsInFamily");
Script script = new Script("params.familyMemberCount / params.totalKidsInFamily");
BucketScriptPipelineAggregationBuilder bs = PipelineAggregatorBuilders
.bucketScript("myScript", bucketsPathsMap, script);
lev1Agg.subAggregation(bs);
aggregation.subAggregation(lev1Agg);
searchSourceBuilder = new SearchSourceBuilder().aggregation(aggregation);
searchSourceBuilder.size(0);
Query built by above code
GET my-alias/_search
{
"size" : 0,
"aggregations" : {
"id" : {
"filter" : {
"terms" : {
"name" : [
"my-name"
],
"boost" : 1.0
}
},
"aggregations" : {
"id" : {
"terms" : {
"field" : "name",
"size" : 1,
"min_doc_count" : 1,
"shard_min_doc_count" : 0,
"show_term_doc_count_error" : false,
"order" : [
{
"_count" : "desc"
},
{
"_term" : "asc"
}
]
},
"aggregations" : {
"familyMemberCount" : {
"sum" : {
"field" : "membersInFamily"
}
},
"totalKidsInFamily" : {
"sum" : {
"field" : "kidsInFamily"
}
},
"myScript" : {
"bucket_script" : {
"buckets_path" : {
"familyMemberCount" : "familyMemberCount",
"totalKidsInFamily" : "totalKidsInFamily"
},
"script" : {
"source" : "params.familyMemberCount / params.totalKidsInFamily",
"lang" : "painless"
},
"gap_policy" : "skip"
}
}
}
}
}
}
}
}