this is my elasticsearch query, which generated by kibana
"aggs": {
"3": {
"terms": {
"field": "policy.weight",
"order": {
"1": "asc"
},
"size": 5
},
"aggs": {
"1": {
"cardinality": {
"field": "item.item_doc_id.keyword"
}
},
"2": {
"terms": {
"field": "policy.name.keyword",
"order": {
"1": "desc"
},
"size": 100
},
"aggs": {
"1": {
"cardinality": {
"field": "item.item_doc_id.keyword"
}
},
"4": {
"terms": {
"field": "policy.description.keyword",
"order": {
"1": "desc"
},
"size": 100
},
"aggs": {
"1": {
"cardinality": {
"field": "item.item_doc_id.keyword"
}
}
}
}
}
}
}
}
}
and this is the same query with java api, which I'm implementing
public BucketList getListOfBucketsTimeRestrictedEV(BucketListInfo bucketListInfo) {
final SearchRequest searchRequest = new SearchRequest(bucketListInfo.getIndexName());
final String field1 = "policy.name.keyword";
final String field2 = "policy.description.keyword";
final String field3 = "policy.weight";
final String field4 = "item.item_doc_id.keyword";
final SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder()
.query(getRangeQueryBuilderWithOptionalFilter(bucketListInfo.getTimestampFieldName(), bucketListInfo.getFilterFieldName(), bucketListInfo.getFilterFieldValue(),
bucketListInfo.getFrom(), bucketListInfo.getTo())).size(100);
TermsAggregationBuilder aggregationBuilder;
aggregationBuilder = AggregationBuilders.terms(field3).field(field3).size(5).order(BucketOrder.count(true));
searchSourceBuilder.aggregation(aggregationBuilder);
aggregationBuilder = AggregationBuilders.terms(field4).field(field4);
searchSourceBuilder.aggregation(aggregationBuilder);
aggregationBuilder = AggregationBuilders.terms(field1).field(field1).size(100);
searchSourceBuilder.aggregation(aggregationBuilder);
aggregationBuilder = AggregationBuilders.terms(field2).field(field2).size(100);
searchSourceBuilder.aggregation(aggregationBuilder);
// aggregationBuilder.subAggregation(AggregationBuilders.terms(subfield).field(subfield));
// searchSourceBuilder.aggregation(aggregationBuilder);
searchRequest.source(searchSourceBuilder);
try {
final SearchResponse response = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
BucketList bucketList = new BucketList();
final Terms terms = response.getAggregations().get(field1);
bucketList.getBuckets().add(terms.getBuckets());
return bucketList;
} catch (Exception e) {
log.error(e.getMessage(), e);
return null;
}
}
Now the question:
Why I'm getting not desired results, say getting more rows than expected?
I'm also using subAggregation here, but I think the subAggregation doesn't work somehow. But I'm debugging and observe that the subAggregation is added in "aggregationBuilder.factoryBuilder container.
Sould I actually use here subAggregation, or just aggregation is enough?
Related
I have a complex and nested JSON File which looks like this
{
"databases": {
"component": "pages/dems/transparency/workspaces/DatasourcesWorkspace.vue",
"children": [
{
"name": "Databases",
"path": "databases",
"component": "pages/dems/transparency/workspaces/Databases.vue",
"hide": true,
"help": "context-sensitive-help/transparency/workspaces/datasources/databases/databases/databases.html"
},
{
"name": "Schemas",
"path": "schemas",
"component": "containers/EmptyContainer.vue",
"hide": true,
"children": [
{
"name": "",
"component": "pages/dems/transparency/workspaces/Schemas.vue",
"hide": true,
"help": "context-sensitive-help/transparency/workspaces/datasources/databases/schemas.html"
},
{
"name": "",
"path": "relationship/:id",
"component": "pages/dems/transparency/workspaces/profilecolumn/ViewRelationShip.vue",
"hide": true,
"help": "context-sensitive-help/transparency/workspaces/relationship/relationship.html"
}
]
}
]
}
}
I want to insert it into MongoDB. The structure would be like this:
{
"_id":"id1",
"field":"databases",
"value":{
"component": "pages/dems/transparency/workspaces/DatasourcesWorkspace.vue",
"children": [
{
"name": "Databases",
"path": "databases",
"component": "pages/dems/transparency/workspaces/Databases.vue",
"hide": true,
"help": "context-sensitive-help/transparency/workspaces/datasources/databases/databases/databases.html"
},
{
"children": [
{
"name": "",
"component": "pages/dems/transparency/workspaces/Schemas.vue",
"hide": true,
"help": "context-sensitive-help/transparency/workspaces/datasources/databases/schemas.html"
},
{
"name": "",
"path": "relationship/:id",
"component": "pages/dems/transparency/workspaces/profilecolumn/ViewRelationShip.vue",
"hide": true,
"help": "context-sensitive-help/transparency/workspaces/relationship/relationship.html"
}
]
}
]
}
}
I need to also check if any object has the containers/EmptyContainer.vue inside the component value then do not insert that object inside MongoDB.
I have tried many methods but didn't work.
Code tried, which is not working as expected:
public ArrayList<Document> processJsonV4(Object source) throws JSONException {
ArrayList<Document> rootDoc = new ArrayList<>();
if (source instanceof JSONObject) {
JSONObject jsonObject = (JSONObject) source;
Document document = new Document();
if (jsonObject.has("component") && !"containers/EmptyContainer.vue".equals(jsonObject.getString("component"))) {
for (Iterator keys = jsonObject.keys(); keys.hasNext(); ) {
String key = (String) keys.next();
Object value = jsonObject.get(key);
document.append(key, value);
}
} else {
for (Iterator keys = jsonObject.keys(); keys.hasNext(); ) {
String key = (String) keys.next();
Object value = jsonObject.get(key);
processJsonV4(value);
}
}
rootDoc.add(document);
}
KLogger.info("RootDoc: " + rootDoc);
return rootDoc;
}
How can I do this using Java?
A simple recursive way could be to construct and add to the list of Documents when you invoke them:
public void readAndConstructMongoDoc() {
JSONObject jsonData = readFileData("<your-file.json>");
List<Document> rootDoc = new ArrayList<>();
constructRootDoc(rootDoc, jsonData);
// Save in mongo -> rootDoc
}
private void constructRootDoc(List<Document> rootDoc, Object source) {
if (source instanceof JSONObject) {
JSONObject jsonObject = new JSONObject();
if (jsonObject.has("component") && !"containers/EmptyContainer.vue".equals(jsonObject.getString("component"))) {
rootDoc.add(convertToMongoDocument(jsonObject));
JSONArray children = jsonObject.getJSONArray("children");
int length = children.length();
for (int i = 0; i < length; i++) {
// Recursively call for all child node and pass the rootDoc
testJson(rootDoc, children.getJSONObject(i));
}
}
}
}
private Document convertToMongoDocument(JSONObject jsonObject) {
Document document = new Document();
document.append("name", jsonObject.getString("name"));
document.append("path", jsonObject.getString("path"));
document.append("hide", Boolean.valueOf(jsonObject.getString("hide")));
document.append("component", jsonObject.getString("component"));
return document;
}
Side Note: I believe your Mongo Document schema can be improvised and doesn't have to be storing the data in this format.
I was new to Elasticsearch and Java. I was trying to convert a nested elasticsearch query which has both MUST and SHOULD functions. Can someone help on this.
GET /list/_search
{
"size": 12,
"query": {
"bool": {
"must": [
{
"bool": {
"should": [
{
"multi_match": {
"query": "city hed",
"type": "bool_prefix",
"fields": [
"cityName":"",
"countryCodeName":"",
"iso":""
]
}
},
{
"multi_match": {
"query": "city hed",
"fuzziness": "AUTO",
"fields": [
"cityName*"
]
}
}
]
}
},
{
"bool": {
"should": [
{
"match": {
"iso": ""
}
},
{
"match": {
"iso": ""
}
}
]
}
}
]
}
}
}
need method logic for it like how it will be in Java API
If you use Java High Client try this:
SearchRequest searchRequest = new SearchRequest("idx_name");
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
MultiMatchQueryBuilder matchQueryBuilder = QueryBuilders
.multiMatchQuery("city hed")
.type(Type.PHRASE_PREFIX)
.field("cityName")
.field("countryCodeName")
.field("iso");
MultiMatchQueryBuilder matchQueryBuilder2 = QueryBuilders
.multiMatchQuery("city hed")
.fuzziness("2")
.field("cityName*");
MatchQueryBuilder matchQuery01 = QueryBuilders.matchQuery("iso", "");
MatchQueryBuilder matchQuery02 = QueryBuilders.matchQuery("iso", "");
BoolQueryBuilder bool01 = QueryBuilders.boolQuery().should(matchQueryBuilder).should(matchQueryBuilder2);
BoolQueryBuilder bool02 = QueryBuilders.boolQuery().should(matchQuery01).should(matchQuery02);
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery().must(bool01).must(bool02);
searchSourceBuilder.query(boolQueryBuilder);
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = getClient().search(searchRequest, RequestOptions.DEFAULT);
I want to implement this json query using ES java client:
{
"query": {
"bool": {
"must_not": [
{
"terms": {
"is_reversed": [
1,
2
]
}
}
],
}
},
"aggs": {
"my_buckets": {
"composite": {
"sources": [
{
"myfield_id": {
"terms": {
"field": "myfield_id",
"missing_bucket": true
}
}
}
],
"size": 65535
},
"aggregations": {
"sum": {
"sum": {
"field": "amount"
}
}
}
}
}
}
I use new elasticsearch-java v8.2 api. it has a few documentation and example.
here's my code:
Query byReveredTransactionType = TermQuery.of(t ->
t.field("is_reversed")
.value(1))._toQuery();
Map<String, CompositeAggregationSource> sources = Map.of("category_id", CompositeAggregationSource.of(b -> b
.terms(t -> t.field("category_id")
.missingBucket(true))));
SearchRequest req = SearchRequest.of(r -> r
.query(q -> q
.bool(b -> b.mustNot(byReveredTransactionType))
)
.aggregations("my_buckets", AggregationBuilders.composite().sources(sources).build()._toAggregation())
.size(Short.MAX_VALUE * 2 - 1)
.build()._toAggregation()
);
here's the code json output:
{
"aggregations": {
"my_buckets": {
"composite": {
"sources": [
{
"myfield_id": {
"terms": {
"field": "myfield_id",
"missing_bucket": true
}
}
}
]
}
}
},
"query": {
"bool": {
"must_not": [
{
"term": {
"is_reversed": {
"value": 1
}
}
}
]
}
},
"size": 65533
}
I have problem in is_reversed part. also I implemented composite and aggregationsparts, but cant combine them in my_buckets.
Thank you!
You need to use terms query and below is way to add aggregation in my_bucket:
List<FieldValue> values = new ArrayList<>();
values.add(FieldValue.of(1));
values.add(FieldValue.of(2));
Query byReveredTransactionType = TermsQuery.of(t -> t.field("is_reversed").terms(v -> v.value(values)))._toQuery();
Map<String, CompositeAggregationSource> cas = new HashMap<>();
cas.put("myfield_id",CompositeAggregationSource.of(c -> c.terms(t -> t.field("myfield_id").missingBucket(true))));
SearchRequest req = SearchRequest.of(r -> r.query(q -> q.bool(b -> b.mustNot(byReveredTransactionType))).aggregations("my_buckets", Aggregation.of(a -> a.composite(c -> c.sources(cas)).aggregations("sum",Aggregation.of(a2 -> a2.sum(s -> s.field("amount")))))));
StringWriter writer = new StringWriter();
JsonGenerator generator = JacksonJsonProvider.provider().createGenerator(writer);
req.serialize(generator, new JacksonJsonpMapper());
generator.flush();
System.out.println(writer.toString());
I want to build the following ES query using BoolQueryBuilders and Aggregator, but I am unable to do that.
{
"size": 0,
"query": {
"bool": {
"should": {
"bool": {
"filter": [
{
"terms": {
"country": [
"France",
"China"
]
}
},
{
"term": {
"lang": "en"
}
}
]
}
}
}
},
"aggs": {
"group_by_country": {
"terms": {
"field": "country",
"size": 0
},
"aggs": {
"top_hits_country": {
"top_hits": {
"size": 1
}
}
}
}
}
}
I am able to build this query without the aggregator, in the following manner -
BoolQueryBuilder innerEntityQueryBuilder = new BoolQueryBuilder();
BoolQueryBuilder queryBuilder = new BoolQueryBuilder()
.filter(QueryBuilders.termsQuery("country", countries))
.filter(QueryBuilders.termQuery("lang", "en"));
innerEntityQueryBuilder.should(queryBuilder);
How do I add the aggregate part as well?
like this：
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("index1");
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.size(0);
BoolQueryBuilder bqb = new BoolQueryBuilder()
.filter(QueryBuilders.termsQuery("country", "France","China"))
.filter(QueryBuilders.termQuery("lang", "en"));
searchSourceBuilder.query(bqb);
TermsAggregationBuilder group = AggregationBuilders
.terms("group_by_country").field("country")
.size(0);
TopHitsAggregationBuilder topHit = AggregationBuilders.topHits("top_hits_country").size(1);
group.subAggregation(topHit);
searchSourceBuilder.aggregation(group);
searchRequest.source(searchSourceBuilder);
I use
ElasticSearchTemplate().queryForPage(SearchQuery, CLASS)
How can I print the full json request?
I manage to print only filter by doing :
searchQuery.getFilter().toString()
But cant manage to do the same with:
searchQuery.getAggregations().toString();
I would like to print in console something like :
"aggs": {
"agg1": {
"terms": {
"field": "basket_id_1",
"size": 0
},
"aggs": {
"basket_id_2": {
"terms": {
"field": "basket_id_2",
"size": 0
},
"aggs": {
"basket_id_3": {
"terms": {
"field": "basket_id_3",
"size": 0
}
}
}
}
}
}
}
This is what I've started using to do the same thing.
{
"top_agg": {
"terms": {
"field": "id",
"size": 100
},
"aggregations": {
"parent": {
"nested": {
"path": "transactions"
},
"aggregations": {
"totals": {
"filter": {
"terms": {
"transactions.type": [
"ttype"
]
}
},
"total_events": {
"cardinality": {
"field": "parent.field"
}
}
}
}
}
}
}
}
NativeSearchQuery query = queryBuilder.build();
if (query.getQuery() != null) {
log.debug(query.getQuery().toString());
}
if (query.getAggregations() != null) {
try {
XContentBuilder builder = XContentFactory.contentBuilder(XContentType.JSON);
builder.startObject();
for (AbstractAggregationBuilder subAgg : query.getAggregations()) {
subAgg.toXContent(builder, ToXContent.EMPTY_PARAMS);
}
builder.endObject();
log.debug(builder.string());
} catch (IOException e) {
log.debug("Error parsing aggs");
}
}
Could you use the SearchResponse.getAggregations().asList() ?