Elasticsearch Java client SearchResponse not recognizing aggregations/buckets - java

Elasticsearch Java client SearchResponse is unable to parse aggregation results. I've come across articles online which suggest adding aggregation types prefixed to the keys. I've added what I thought applies in my use case, such as "sterms# and sum#" but I am unable to figure out which type applies to the main filter (key:'matched' in my case). I am expecting the buckets object to be populated but it is currently being returned as an empty array despite response from elasticsearch containing aggregations.
Note: This is to be able to unit test.
Json response:
{
"took": 7,
"timed_out": false,
"_shards": {
"total": 75,
"successful": 75,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 776329,
"max_score": 0,
"hits": []
},
"aggregations": {
"sterms#matched": {
"doc_count": 15,
"sterms#id": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "mykey",
"doc_count": 15,
"sterms#manufacturerName": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "subkey",
"doc_count": 15
}
]
},
"sum#totalCars": {
"value": 214244
},
"sum#totalTrucks": {
"value": 155738
}
}
]
}
}
}
}
Parser in Java
public static SearchResponse getSearchResponseFromJson(final String jsonFileName) {
try {
String jsonResponse = IOUtils.toString(new FileInputStream(jsonFileName));
NamedXContentRegistry registry = new NamedXContentRegistry(getDefaultNamedXContents());
XContentParser parser = JsonXContent.jsonXContent.createParser(registry, jsonResponse);
return SearchResponse.fromXContent(parser);
} catch (IOException e) {
throw new RuntimeException("Failed to get resource: " + jsonFileName, e);
} catch (Exception e) {
throw new RuntimeException(("exception " + e));
}
}
private static List<NamedXContentRegistry.Entry> getDefaultNamedXContents() {
Map<String, ContextParser<Object, ? extends Aggregation>> map = new HashMap<>();
map.put(TopHitsAggregationBuilder.NAME, (parser, content) -> ParsedTopHits.fromXContent(parser, (String) content));
map.put(StringTerms.NAME, (parser, content) -> ParsedStringTerms.fromXContent(parser, (String) content));
map.put(SumAggregationBuilder.NAME, (parser, content) -> ParsedSum.fromXContent(parser, (String) content));
//map.put(ScriptedMetricAggregationBuilder.NAME, (parser, content) -> ParsedScriptedMetric.fromXContent(parser, (String) content));
List<NamedXContentRegistry.Entry> entries = map.entrySet()
.stream()
.map(entry -> new NamedXContentRegistry.Entry(Aggregation.class,
new ParseField(entry.getKey()),
entry.getValue()))
.collect(Collectors.toList());
return entries;
}
Parsed response with empty buckets (expectation is for the buckets to have aggregations totalCars and totalTrucks)
{
"took": 7,
"timed_out": false,
"_shards": {
"total": 75,
"successful": 75,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 776329,
"max_score": 0,
"hits": []
},
"aggregations": {
"sterms#matched": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": []
}
}
}
I am not sure if sterms is the right type for the filter


The answer for what type needs to be used is 'filter' (filter#matched). And Java class counterpart is FilterAggregationBuilder

Related

Elasticsearch Java 8.5: retrieve all Ids

I am trying to retrieve all (elasticsearch) Ids of an index using the following code.
SearchRequest sr = SearchRequest.of(r -> r
.index("my_index")
.source(s -> s.fetch(false))
.size(10000));
System.out.println("Request: " + sr);
final SearchResponse<MyDoc> response;
try {
response = this.esClient.search(sr, MyDoc.class);
} catch (ElasticsearchException | IOException e) {
throw new MyException("fetch all ids: request failed: " + e.getMessage(), e);
}
System.out.println("Response: " + response);
There are no results in the response. However, the printed request is
POST /my_index/_search?typed_keys=true {"_source":false,"size":10000}
which works perfectly fine when run directly as a REST request.
Any idea how to do it using the Java client?
The REST response is
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 999,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "my_index",
"_id": "2LHJSP6dTjXuEM2vsEDyxdG4Y7HPzXL15tFvrkyZm8xn",
"_score": 1
},
{
"_index": "my_index",
"_id": "A8RCf2mV4qeWvLxSfzKNX418E734uEifCenoCAiM3syB",
"_score": 1
},
...
]
}
}

Maybe this works for you:
SearchRequest sr = SearchRequest.of(r -> r
.index("idx_name")
.source(s -> s.fetch(false))
.size(10000));
System.out.println("Request: " + sr);
try {
var response = this.esClient.search(sr, Void.class);
response.hits().hits().forEach(hit -> {
System.out.println("ID: " + hit.id());
});
} catch (ElasticsearchException | IOException e) {
}

Elasticsearch Rest High Level Client aggregate fields dynamically

I am trying to generate query dynamically based on the inputs but in the generated query i can see there are only two aggregations are getting generated how can i make each fields to have the separate aggregations below is the code what i have tried and the response what i'm getting.
From main() i'm calling
buildSearchCriteria("1");
Here i am setting the aggregation type and respective values:
public static void buildSearchCriteria(String... exceptionId) {
SearchCriteria searchCriteria = new SearchCriteria();
Map<String, List<FieldNameAndPath>> stringListMap = new HashMap<>();
stringListMap.put("nested", asList(new FieldNameAndPath("nested", "recommendations",
"recommendations", null, emptyList(), 1)));
stringListMap.put("filter", asList(new FieldNameAndPath("filter", "exceptionIds", "recommendations.exceptionId.keyword",
asList(exceptionId),
asList(new NestedAggsFields("terms", "exceptionIdsMatch")), 2)));
stringListMap.put("terms", asList(new FieldNameAndPath("terms", "by_exceptionId", "recommendations.exceptionId.keyword", null, emptyList(), 3),
new FieldNameAndPath("terms", "by_item", "recommendations.item.keyword", null, emptyList(), 4),
new FieldNameAndPath("terms", "by_destination", "recommendations.location.keyword", null, emptyList(), 5),
new FieldNameAndPath("terms", "by_trans", "recommendations.transportMode.keyword", null, emptyList(), 6),
new FieldNameAndPath("terms", "by_sourcelocation", "recommendations.sourceLocation.keyword", null, emptyList(), 7),
new FieldNameAndPath("terms", "by_shipdate", "recommendations.shipDate", null, emptyList(), 8),
new FieldNameAndPath("terms", "by_arrival", "recommendations.arrivalDate", null, emptyList(), 9)));
stringListMap.put("sum", asList(new FieldNameAndPath("sum", "quantity", "recommendations.transferQuantity", null, emptyList(), 10),
new FieldNameAndPath("sum", "transfercost", "recommendations.transferCost", null, emptyList(), 11),
new FieldNameAndPath("sum", "revenueRecovered", "recommendations.revenueRecovered", null, emptyList(), 12)));
System.out.println(stringListMap);
searchCriteria.setStringListMap(stringListMap);
aggregate(searchCriteria);
}
Below is the aggregate function which will get the the above information and builds query:
public static void aggregate(SearchCriteria searchCriteria) throws IOException {
Map<String, List<FieldNameAndPath>> map = searchCriteria.getStringListMap();
List<FieldNameAndPath> nesteds = map.get("nested");
List<FieldNameAndPath> filter = map.get("filter");
List<FieldNameAndPath> terms = map.get("terms");
List<FieldNameAndPath> sums = map.get("sum");
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
AggregationBuilder aggregationBuilder = new SamplerAggregationBuilder("parent");
nesteds.stream().forEach(l -> buildAggregations(l, aggregationBuilder));
filter.stream().forEach(l -> buildAggregations(l, aggregationBuilder));
terms.stream().forEach(l -> buildAggregations(l, aggregationBuilder));
sums.stream().forEach(l -> buildAggregations(l, aggregationBuilder));
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("index");
searchRequest.types("type");
sourceBuilder.aggregation(aggregationBuilder);
searchRequest.source(sourceBuilder);
System.out.println(searchRequest.source().toString());
}
buildAggregations method:
private static AggregationBuilder buildAggregations(FieldNameAndPath fieldNameAndPath , AggregationBuilder parentAggregationBuilder) {
if(fieldNameAndPath.getAggType().equals("nested")){
parentAggregationBuilder = AggregationBuilders.nested(fieldNameAndPath.getFieldName(), fieldNameAndPath.getFieldPath());
}
if(fieldNameAndPath.getAggType().equals("filter")){
parentAggregationBuilder.subAggregation(AggregationBuilders
.filter(fieldNameAndPath.getFieldName(),
QueryBuilders.termsQuery(fieldNameAndPath.getNestedAggs()
.stream().map(nestedAggsFields -> nestedAggsFields.getFieldName()).findFirst().get(), fieldNameAndPath.getFieldValues())));
}
if(fieldNameAndPath.getAggType().equals("terms")){
parentAggregationBuilder.subAggregation(AggregationBuilders.terms(fieldNameAndPath.getFieldName())
.field(fieldNameAndPath.getFieldPath()));
}
if(fieldNameAndPath.getAggType().equals("sum")){
parentAggregationBuilder.subAggregation(AggregationBuilders.
sum(fieldNameAndPath.getFieldName()).field(fieldNameAndPath.getFieldPath()));
}
return parentAggregationBuilder;
}
SearchCriteria class:
#Data
public class SearchCriteria {
Map<String, List<FieldNameAndPath>> stringListMap;
private List<String> searchFields;
}
And the DTO FieldNameAndPath:
public class FieldNameAndPath{
private String aggType;
private String fieldName;
private String fieldPath;
private List<String> fieldValues;
private List<NestedAggsFields> nestedAggs;
private int order;
}
And the query output from the above code is:
{
"aggregations": {
"parent": {
"sampler": {
"shard_size": 100
},
"aggregations": {
"exceptionIds": {
"filter": {
"terms": {
"exceptionIdsMatch": [
"1"
],
"boost": 1
}
}
},
"by_exceptionId": {
"terms": {
"field": "recommendations.exceptionId.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
}
},
"by_item": {
"terms": {
"field": "recommendations.item.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
}
},
"by_destination": {
"terms": {
"field": "recommendations.location.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
}
},
"by_trans": {
"terms": {
"field": "recommendations.transportMode.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
}
},
"by_sourcelocation": {
"terms": {
"field": "recommendations.sourceLocation.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
}
},
"by_shipdate": {
"terms": {
"field": "recommendations.shipDate",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
}
},
"by_arrival": {
"terms": {
"field": "recommendations.arrivalDate",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
}
},
"quantity": {
"sum": {
"field": "recommendations.transferQuantity"
}
},
"transfercost": {
"sum": {
"field": "recommendations.transferCost"
}
},
"revenueRecovered": {
"sum": {
"field": "recommendations.revenueRecovered"
}
}
}
}
}
}
Expected Query is:
{
"size": 0,
"aggregations": {
"exceptionIds": {
"nested": {
"path": "recommendations"
},
"aggregations": {
"exceptionIdsMatch": {
"filter": {
"terms": {
"recommendations.exceptionId.keyword": [
"1"
],
"boost": 1
}
},
"aggregations": {
"by_exceptionId": {
"terms": {
"field": "recommendations.exceptionId.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
},
"aggregations": {
"by_item": {
"terms": {
"field": "recommendations.item.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
},
"aggregations": {
"by_destination": {
"terms": {
"field": "recommendations.location.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
},
"aggregations": {
"by_trans": {
"terms": {
"field": "recommendations.transportMode.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
},
"aggregations": {
"by_sourcelocation": {
"terms": {
"field": "recommendations.sourceLocation.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
},
"aggregations": {
"by_shipdate": {
"terms": {
"field": "recommendations.shipDate",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
},
"aggregations": {
"by_arrival": {
"terms": {
"field": "recommendations.arrivalDate",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
},
"aggregations": {
"quantity": {
"sum": {
"field": "recommendations.transferQuantity"
}
},
"transfercost": {
"sum": {
"field": "recommendations.transferCost"
}
},
"revenueRecovered": {
"sum": {
"field": "recommendations.revenueRecovered"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}

Kibana scripted field - text to timestamp conversion parse exception

I am unable to convert a text field ts1 in my ES index (sample value 2017-09-30 04:53:39.412496Z) to timestamp by using scripted fields in Kibana. The SimpleDateFormat expression works in Java, just not in Elasticsearch. This mentions a similar problem but the solution doesn't work for me.
Query
GET <index_name>/_search
{
"query" : {
"match_all": {}
},
"script_fields" : {
"test1" : {
"script" : {
"lang": "painless",
"source": """new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSSSSS'Z'").parse(doc['ts1'].value).getTime()"""
}
}
}
}
Output
{
"took": 76,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 4,
"skipped": 0,
"failed": 1,
"failures": [
{
"shard": 3,
"index": "<index_name>",
"node": "-TjsxK4-QsqgLQRQ-DSKGQ",
"reason": {
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"java.text.DateFormat.parse(DateFormat.java:366)",
"""new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSSSSS'Z'").parse(doc['ts1'].value).getTime()""",
" ^---- HERE"
],
"script": """new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSSSSS'Z'").parse(doc['ts1'].value).getTime()""",
"lang": "painless",
"caused_by": {
"type": "parse_exception",
"reason": """Unparseable date: "04""""
}
}
}
]
},
"hits": {
"total": 1,
"max_score": 1,
"hits": []
}
}
Java Code that works -
import java.text.SimpleDateFormat;
public class Main{
public static void main(String []args){
String text2 = "2017-09-30 04:53:39.123123Z";
SimpleDateFormat s2 = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSSSSS'Z'");
try {
System.out.println(s2.parse(text2).getTime());
} catch (Exception e) {
System.out.println(e);
}
}
}

Unable to extract out name in 4 level deep nested JSON array

I am trying to create android app that shows details about the restaurants. I have parseJSON function which takes string json as an argument. What I am trying to do for now is to show name of individual restaurants in the json below. I am able to extract individual restaurant(3 levels deep) and print them out through log.d console but I am not able to extract out the individual restaurant's name. Here is the code
public void parseJSON(String jsonQuery){
//Parsing JSON
try {
JSONObject jsonObject = new JSONObject(jsonQuery);
if (jsonObject != null){
JSONArray restaurants = jsonObject.getJSONArray("restaurants");
for (int i = 0; i < restaurants.length(); i++){
JSONObject restaurant = restaurants.getJSONObject(i);
if (restaurant != null){
String tempName = restaurant.getString("name");
Log.d(TAG, "Restaurant name: " + tempName);
}
}
}
} catch (JSONException e) {
Log.d(TAG, "Exception catched: " + e);
e.printStackTrace();
}
}
Here is my json file.
{
"results_found": 1281966,
"results_start": 0,
"results_shown": 2,
"restaurants": [
{
"restaurant": {
"R": {
"res_id": 16607974
},
"apikey": "123456789",
"id": "16607974",
"name": "Bassine Specialty Cheese",
"url": "https://www.zomato.com/bass-vic/bassine-specialty-cheese-bass?utm_source=api_basic_user&utm_medium=api&utm_campaign=v2.1",
"location": {
"address": "2125 Bass Hwy",
"locality": "Bass",
"city": "Bass",
"city_id": 1341,
"latitude": "-38.4833750000",
"longitude": "145.4670320000",
"zipcode": "3995",
"country_id": 14,
"locality_verbose": "Bass, Bass"
},
"switch_to_order_menu": 0,
"cuisines": "Ice Cream",
"average_cost_for_two": 7,
"price_range": 1,
"currency": "$",
"offers": [],
"thumb": "",
"user_rating": {
"aggregate_rating": "2.9",
"rating_text": "Average",
"rating_color": "FFBA00",
"votes": "5"
},
"photos_url": "https://www.zomato.com/bass-vic/bassine-specialty-cheese-bass/photos?utm_source=api_basic_user&utm_medium=api&utm_campaign=v2.1#tabtop",
"menu_url": "https://www.zomato.com/bass-vic/bassine-specialty-cheese-bass/menu?utm_source=api_basic_user&utm_medium=api&utm_campaign=v2.1&openSwipeBox=menu&showMinimal=1#tabtop",
"featured_image": "",
"has_online_delivery": 0,
"is_delivering_now": 0,
"deeplink": "zomato://restaurant/16607974",
"has_table_booking": 0,
"events_url": "https://www.zomato.com/bass-vic/bassine-specialty-cheese-bass/events#tabtop?utm_source=api_basic_user&utm_medium=api&utm_campaign=v2.1",
"establishment_types": []
}
},
{
"restaurant": {
"R": {
"res_id": 17649424
},
"apikey": "3d93604b4a84d85f374f39ea3b644132",
"id": "17649424",
"name": "Schobels' Restaurant",
"url": "https://www.zomato.com/columbus-tx/schobels-restaurant-columbus?utm_source=api_basic_user&utm_medium=api&utm_campaign=v2.1",
"location": {
"address": "2020 Milam St 78934",
"locality": "Columbus",
"city": "Columbus",
"city_id": 9241,
"latitude": "29.6965000000",
"longitude": "-96.5405000000",
"zipcode": "78934",
"country_id": 216,
"locality_verbose": "Columbus, Columbus"
},
"switch_to_order_menu": 0,
"cuisines": "German, Southern",
"average_cost_for_two": 25,
"price_range": 2,
"currency": "$",
"offers": [],
"thumb": "",
"user_rating": {
"aggregate_rating": "4.0",
"rating_text": "Very Good",
"rating_color": "5BA829",
"votes": "164"
},
"photos_url": "https://www.zomato.com/columbus-tx/schobels-restaurant-columbus/photos?utm_source=api_basic_user&utm_medium=api&utm_campaign=v2.1#tabtop",
"menu_url": "https://www.zomato.com/columbus-tx/schobels-restaurant-columbus/menu?utm_source=api_basic_user&utm_medium=api&utm_campaign=v2.1&openSwipeBox=menu&showMinimal=1#tabtop",
"featured_image": "",
"has_online_delivery": 0,
"is_delivering_now": 0,
"deeplink": "zomato://restaurant/17649424",
"has_table_booking": 0,
"events_url": "https://www.zomato.com/columbus-tx/schobels-restaurant-columbus/events#tabtop?utm_source=api_basic_user&utm_medium=api&utm_campaign=v2.1",
"establishment_types": []
}
}
]
}
Error which I am getting:
org.json.JSONException: No value for name

Your jsonQuery contain JsonObject inside JsonObject named restaurant so you need to get JsonObject from JSONObject restaurant = restaurants.getJSONObject(i);
Like JSONObject restaurantsJSONObject =restaurant.getJSONObject("restaurant");
Try below example to get name from object
try {
JSONObject jsonObject = new JSONObject(jsonQuery);
if (jsonObject != null){
JSONArray restaurants = jsonObject.getJSONArray("restaurants");
for (int i = 0; i < restaurants.length(); i++){
JSONObject restaurant = restaurants.getJSONObject(i);
JSONObject restaurantsJSONObject =restaurant.getJSONObject("restaurant");
if(restaurantsJSONObject.has("name")) {
String tempName = restaurantsJSONObject.getString("name");
Log.d(TAG, "Restaurant name: " + tempName);
}
}
}
} catch (JSONException e) {
Log.d(TAG, "Exception catched: " + e);
e.printStackTrace();
}

How do elasticsearch aggreation return extra field

How do use elasticsearch terms aggreation get a extra field value? This is my query string
"aggs":{
"group_by_ruleSets":{
"terms":{
"size":100,
"field":"alert.ruleSets",
"order":{
"_count":"desc"
}
}
}
}
This is elasticsearch return result:
"aggregations": { "terms_group_by": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "10457", "doc_count": 2 } ] } } }
and this is the result I want:
"aggregations": { "terms_group_by": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "10457", "doc_count": 2 "setName":"Test RuleSet" } ] } } }
The field "setName" has in elasticsearch.

Categories

Resources